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Abstract 


Synthetic Vision Systems and Enhanced Flight Vision System 
(SVS/EFVS) technologies have the potential to provide additional 
margins of safety for aircrew performance and enable operational 
improvements for low visibility operations in the terminal area 
environment with equivalent efficiency as visual operations. To meet this 
potential , research is needed for effective technology development and 
implementation of regulatory standards and design guidance to support 
introduction and use of SVS/EFVS advanced cockpit vision technologies 
in Next Generation Air Transportation System (NextGen) operations. 

A fixed-base pilot-in-the-loop simulation test was conducted at NASA 
Langley Research Center that evaluated the use of SVS/EFVS in NextGen 
low visibility approach and landing operations. Twelve crews flew 
approach and landing operations in a simulated NextGen Chicago 
O'Hare environment. Various scenarios tested the potential for using 
EFVS to conduct approach, landing, and roll-out operations in visibility 
as low as 1000 feet runway visual range (RVR). Also, SVS was tested to 
evaluate the potential for lowering decision heights (DH) on certain 
instrument approach procedures below what can be flown today. 

Expanding the portion of the visual segment in which EFVS can be used 
in lieu of natural vision from 100 feet above the touchdown zone 
elevation to touchdown and rollout in visibilities as low as 1000 feet 
RVR appears to be viable as touchdown performance was acceptable 
without any apparent workload penalties. A lower DH of 150 feet and/or 
possibly reduced visibility minima using SVS appears to be viable when 
implemented on a Head-Up Display, but the landing data suggests 
further study for head-down implementations. 

1 Introduction 

The U.S. air transportation system is undergoing a transformation to accommodate the movement of 
large numbers of people and goods in a safe, efficient, and reliable manner [1]. One of the key 
capabilities envisioned to achieve this Next Generation Air Transportation System (NextGen) is the 
concept of equivalent visual operations (EVO). EVO is the capability to achieve the safety of 
current-day Visual Flight Rules (VFR) operations and maintain the operational tempos of VFR 
irrespective of the weather and visibility conditions. 

One research challenge for EVO is the definition of required equipage on the aircraft and at the 
airport. With today’s equipment and regulations, significant investment is required in on-board 
equipment for navigation, surveillance, and flight control and on the airport for precision guidance 
systems and approach lighting systems for “all-weather” landing capability [2]. The levels of 
equipment redundancy, capability, maintenance, performance and crew training dramatically increase 
as landing visibility minima decrease. Synthetic Vision Systems and Enhanced Flight Vision 
Systems (SVS/EFVS) offer a means of providing EVO capability without significant airport 
infrastructure investment while potentially increasing efficiency and throughput during low visibility 
operations. 
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NASA Langley Research Center (NASA LaRC) and the Department of Transportation/Federal 
Aviation Administration (DOT/FAA) are jointly conducting collaborative research to ensure effective 
technology development and implementation of regulatory standards and design guidance to support 
the introduction and use of SVS/EFVS advanced cockpit vision technologies in NextGen operations. 
These technologies have the potential to enable operational improvements that would benefit low 
visibility surface, arrival, and departure operations in the terminal environment with equivalent 
efficiency as visual operations. This work builds from and extends the current operational use and 
certification of existing SVS/EFVS technologies toward all-weather, low visibility operations for 
NextGen. 

In addition, under NASA’s Aviation Safety Program, research is being conducted to evaluate the 
influence of Cockpit Display of Traffic Information (CDTI) and SVS/EFVS technologies on a pilot’s 
situation and traffic awareness during low visibility surface operations. This research is motivated in 
part by the FAA’s 2010 Annual Runway Safety Report [3], which identifies planned mid-term (2012- 
2018) NextGen research initiatives that include the use of and integration of CDTI and SVS/EFVS 
technologies. As described in this FAA report, under low visibility operations, “Location information 
of aircraft and vehicles on the airport surface will be displayed on moving maps using Cockpit 
Display of Traffic Information (CDTI) or aided by Enhanced Flight Vision Systems (EFVS), 
Enhanced Vision Systems (EVS), Synthetic Vision Systems (SVS), or other types of advanced vision 
or virtual vision technology.” 



2 Background 


S VS is a computer-generated image of the external scene topography, generated using aircraft 
attitude, high-precision navigation, and data of the terrain, obstacles, cultural features, and other 
required flight information. EFVS is an electronic means to provide a display (typically on a head-up 
display, or HUD) of the external scene by use of an imaging sensor, such as a Forward-Looking 
InfraRed (FLIR) or millimeter wave radar. Both SVS and EFVS are “vision-based” technologies 
intended to create, supplement, or enhance the natural vision of the pilot. 

NASA and others have developed and shown SVS technologies that provide significant 
improvements in terrain awareness and reductions for the potential of Controlled-Flight-Into-Terrain 
incidents/accidents [4-6], improvements in Flight Technical Error (FTE) to meet Required Navigation 
Performance criteria [7,8], and improvements in Situation Awareness (SA) without concomitant 
increases in workload compared to current generation cockpit technologies [9-13]. As such, SVS, 
often displayed on a Head-Down Display (HDD), is emerging as standard equipage for Part 23 and 
Part 25 business and General Aviation (GA) aircraft flight decks even though, to date, no “operational 
credit” is obtained by SVS equipage [14]. Operational credit is a specific benefit afforded the aircraft 
operator from application of FAA Advisory Circulars. 

EFVS capability on a HUD using FLIR sensor technology has garnered a significant share of the 
business aircraft market and is growing in Part 121 and 135 operations [15]. EFVS provides many of 
the same operational benefits as SVS technology, but it uses a real-time view of the external 
environment, independent of the aircraft navigation solution or database. These differences, in part, 
enable operational credit by use of an approved EFVS. In 2004, Title 14 of the Code of Federal 
Regulations (CFR) Section (§) 91.175 was amended to enable operators conducting straight-in 
instrument approach procedures (in other than Category II or Category III operations) to descend 
below the published Decision Altitude (DA), Decision Height (DH) or Minimum Descent Altitude 
(MDA) down to 100 feet (ft) above the touchdown zone elevation (TDZE) using an approved EFVS 
in lieu of natural vision. The enhanced flight visibility provided by the EFVS must meet or exceed 
the published visibility for the approach being flown and the required visual references to descend 
from the DA/DH/MDA to 100 ft above the TDZE must be in view on the EFVS. An approved EFVS 
must meet the requirements of § 9 1.1 75 (m) and must be presented on a HUD or an equivalent head- 
up display that might be found acceptable to the FAA. In order to descend below 100 feet above the 
touchdown zone elevation, natural vision must be used. 

Synthetic and Enhanced Vision Systems (SEVS) technologies, such as SVS/EFVS in combination 
with HDD/HUD, form the basis for an electronic display of visual flight references (terrain, obstacles, 
and operations-critical navigational and situational references) on electronic cockpit display(s) for the 
flight crew. Integrating these SEVS displays with conformal symbology provides important situation, 
guidance, and/or command information as necessary and/or appropriate to enable all weather 
approach and landing operations. The primary reference for maneuvering the airplane is based on 
what the pilot sees through the SEVS, in lieu of or supplemental to the pilot’s natural vision, in low 
visibility conditions. 

The key concept of the revisions to 14 CFR§ 91.175 is that an EFVS can be used in lieu of natural 
vision from the DA/DH/MDA to 100 ft height above the TDZE provided the enhanced vision image 
in the HUD meets or exceeds the published visibility required for the approach being flown and 
required visual references are in view. Minimum aviation system performance standards are now 
available in Radio Technical Commission for Aeronautics (RTCA) DO-315 [16]. In addition, FAA 
Advisory Circular AC 20-167 [17] provides guidance on certification and installation of EFVS and 
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FAA Advisory Circular AC 90-106 [18] provides guidance for obtaining operational approval to use 
EFVS in lieu of natural vision to descend below DA/DH or MDA. RTCA DO-3 15 also provides 
performance standards for SVS but with no additional operational credit. In other words, installing 
SVS does not change the airplane’s existing operational capability. It can only be used for Situation 
Awareness. 

The emerging challenge for NextGen, and the subject of NASA research, is to develop performance- 
based standards for SEVS technologies that create EVO. The first part of this challenge is the 
development of performance-based standards that could lead to future operational approvals for 
SVS/EFVS. The second part of this challenge is to evaluate the influence of SEVS technologies and 
other emerging flight-deck information sources such as CDTI and their potential use to achieve EVO. 

The design and use of SEVS technology is integrally tied to the operating rules for landing and take- 
off under Instrument Flight Rules (IFR) (14 CFR § 91.175). For EFVS operations, § 91.175 was 
modified such that, to operate below the DA/DH or MDA, the pilot must determine that the enhanced 
flight visibility is not less than that published for the instrument approach being used and certain 
visual references must be seen using the EFVS to continue the descent below the published DA/DH 
or MDA (see 14 CFR § 91.175). No lower than 100 ft height above the TDZE, the visual references 
that positively identify the runway of intended landing must be distinctly visible and identifiable 
using natural vision. 

The FAA started a rulemaking project to expand operational credit for EFVS beyond what is 
currently authorized by the regulations (under 14 CFR § 91.175) [15, 19]. RTCA DO-315A [20] was 
drafted to establish performance standards in concert with this rulemaking project. Minimum system 
performance standards are now published for EFVS operations through the approach to touchdown in 
visibility as low as 1000 ft runway visual range (RVR) by sole use of an approved EFVS in lieu of 
natural vision. Simply stated, (in the RTCA DO-3 15A MASPS) the visual segment of the approach 
can now be accomplished by using either enhanced flight visibility or natural vision. Past NASA 
research [21] supports the viability of this expanded EFVS operational credit where it was shown that 
using an EFVS to hand fly approaches through touchdown resulted in excellent localizer tracking 
performance (less than 1/3 dot localizer deviation between 300 ft and 100 ft HAT) and an 
improvement in glideslope tracking performance over hand-flown EFVS approaches to touchdown 
where natural vision, not enhanced vision, was required to positively identify the runway of intended 
landing. 

The joint RTCA SC-213 and European Organisation for Civil Aviation Equipment (EUROCAE) 
Working Group 79 committee also drafted RTCA DO-315B [22] to establish minimum performance 
standards for possible operational credit for SVS. Unlike EFVS, the possible path for operational 
credit is not through revision of 14 CFR § 91.175, but is based on FAA Order 8400.13 (“Procedures 
for the Evaluation and Approval of Facilities for Special Authorization Category I Operations and All 
Category II and III Operations”). Specifically, RTCA DO-315B establishes performance standards 
for SVS enabling lower than standard Category I minima or a reduction in the required minimum 
visibility. These RTCA DO-315B performance standards for SVS operational credit do not require 
the use of a HUD. 

A fixed-base experiment was conducted to evaluate selected elements of the proposed performance 
standards for expanded EFVS and SVS operational credits. Specifically, the high-level objectives of 
this simulation test were to: 

• Evaluate the operational feasibility, pilot workload, and pilot acceptability of conducting a 
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straight-in instrument approach procedure with published vertical guidance using EFVS (i.e., 
FLIR imagery on a HUD) for the approach, landing, roll-out and turn-off in weather and 
visibility as low as 1000 ft RVR. 

• Evaluate the operational feasibility, pilot workload, and pilot acceptability of conducting a 
straight-in instrument approach to a 150 ft DA/DH procedure with published vertical 
guidance using SVS (displayed either head-down or head-up) and to transition to natural out- 
the-window (OTW) visual conditions for landing in weather and visibility as low as 1400 ft 
RVR. 

In addition, NASA-specific test objectives included: 

• Evaluation of time required, accuracy, and pilot workload associated with recognizing and 
reacting to potential ground collisions or conflicts with other aircraft, vehicles and 
obstructions across a range of visibility and lighting conditions using an EFVS and SVS. 

• Evaluation of the effect OTW visual cue alerting by the pilot monitoring (PM) has on the 
performance and visual attention of the pilot- flying (PF) during approach and landing low 
visibility operations. This assessment was conducted by modulating the crew complement 
(i.e., single versus dual pilot operations) using SEVS technologies. This objective enables 
correlation and comparison to a follow-on (single pilot) flight test configuration. These runs 
were not conducted to advocate nor imply the possible acceptance of single pilot operations 
for Part 25 aircraft. 

• Examination of CDTI influences on the time required, accuracy, and pilot workload 
associated with recognizing and reacting to potential ground collisions or conflicts with other 
aircraft, vehicles and obstructions when using EFVS and SVS. 
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3 Methodology 

3.1 Subjects 

Twenty-four pilots served as test subjects for the research, representing twelve flight crews. Ten 
crews flew for major U.S. air carriers and were paired by airline to ensure crew coordination and 
cohesion with regard to terminal and surface operational procedures. The other crews were business 
aircraft operators, flying Gulfstream G-V or G450 aircraft equipped with EFVS and SVS. All test 
subjects were male. The Captains’ average age was 55 years and the First Officers’ average age was 
47.5 years. The Captains had an average of over 14,661 flight hours with 21 years of commercial 
flying. The First Officers had an average of over 10,648 flight hours with 14 years of commercial 
experience. The Captains were recruited on the basis of HUD experience (at least 100 hours), with 
preference given to pilots with Enhanced Vision (EV)/EFVS experience. All pilots were required to 
hold an Airline Transport Pilot rating. 

The Captain was the designated pilot-flying (PF) throughout all the trials and the First Officer served 
as the pilot-monitoring (PM). 

3.2 Simulation Facility 

This research was conducted in the Research Flight Deck (RFD) simulator at NASA FaRC (Figure 
1). The RFD is configured with four 10.5-inch Vertical (V) by 13.25-inch Horizontal (H), 1280x1024 
pixel resolution color displays, tiled across the instrument panel. Also, the RFD includes a HUD on 
the left side of the cab, Mode Control Panel, Flight Management System (FMS), and two Electronic 
Flight Bags (EFBs) (Figure 1). 

The full-mission RFD simulates a modem twin-jet transport aircraft. The cab is populated with flight 
instmmentation and pilot controls, including the overhead subsystem panels, to replicate the 
simulated aircraft. A NASA-designed sidestick control system is used in this cab instead of 
conventional wheel and column for manual flight control. 

A collimated OTW scene is produced through five Barco model 7 120 A projectors shown on a 10 foot 
diameter SEOS panorama mirror cell using a 5-channel EP-1000 image generation system providing 
approximately 200°H by 40°V field-of-view (FOV) at 26 pixels per degree. 

The sidestick inceptor force gradients and deflection characteristics mimic the Airbus 320 aircraft 
[23]. A parabolic shaping gearing was used between the normalized stick deflection of the sidestick 
and the elevator and aileron commands of the simulated aircraft. The parabolic shaping provided 
acceptable handling characteristics for the approach, landing, and departure tasks. The pilot and co- 
pilot inceptors are directly linked as if mechanically connected. 

The auto-throttle system backdrives the throttle handles to directly reflect the power setting 
commanded to the engines. Take-off, go-around (TOGA) buttons and autothrottle disconnect buttons 
were located on the throttle handles. 
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Figure 1. Research Flight Deck Simulator with HUD, Head-down Instrument Panel, and EFB 


3.2.1 Simulator Database 

Operations were simulated at Chicago O’Hare International Airport (FAA identifier: ORD). The 
simulation was built around FAA source data for ORD, valid from 1 1 March 2010 to 8 April 2010. 
These data were used to develop all flight plans, scenarios, approach paths, and OTW, synthetic 
vision (SV) and EV databases. 

Day simulations were flown, with the weather tailored to create the desired visibility conditions. 

Approaches were flown only to runways with Medium intensity Approach Lighting System with 
Runway alignment indicator lights (MALSR) installed. Testing included an experimental variation of 
touchdown zone and centerline (TDZ/CL) lights (on and off), where operations with TDZ/CL lights 
were conducted on ORD Runway 9R; otherwise, ORD Runways 4R, 22L, or 22R were used. All 
runways included high intensity runway lights and serviceable centerline and surface markings. 
Airport lighting was drawn using calligraphics. 

3.2.2 Audio Effects 

The RFD simulator included standard audio effects representative of current day air carriers. Of 
particular importance, altitude call-outs were played over the flight deck speakers. The automatic 
altitude calls-out started at “500 feet” when the aircraft was 500 ft above the TDZE. The 
“approaching minimums” and “minimums” call-outs were generated at 100 ft above and at the 
planned DA/DH for a given run. 

Flare “prompts” in the form of additional altitude call-outs were used on all runs (“100,” “50,” “40,” 
“30,” “20,” and “10” at the corresponding radar altitudes in feet). 

3. 2. 3 Head-Down Displays 

Figure 1 shows the simulator’s four main instrument panel displays on the HDD Panel: a) PF left 
display, including primary flight display (PFD); b) PF right display including navigation display 
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(ND); c) PM left display, including ND; and, d) PM right display, including PFD. The format and 
content of these displays were varied experimentally as described in Section 3.7.1. 

3. 2. 4 Head- Up Display 

The RFD is equipped with a Rockwell-Collins Head-up Guidance System (HGS™)-4000 HUD. The 
HUD is collimated and subtends approximately 26° H by 21° V FOV. The HUD projects the imagery 
from a Cathode Ray Tube source in a stroke-and-raster format (Figure 2). The raster input to the 
HUD was either a SV or EV source imagery in an RS-343 format [24]. Note that to maintain 
conformality with the outside world, the FOV of the HUD raster image was fixed and could not be 
varied by the pilot. The stroke symbology format was a modified version of the HGS-4000 primary 
mode format, with the compass rose symbol removed (Figure 3). The stroke symbology included a 
mnway outline (edge lines) which was conformally drawn using a 8000-ft x 200-ft runway until 50 ft 
above field level (AFL), a flight path marker, a flight path angle reference cue and a flight-path 
referenced guidance cue (Figure 3). The guidance cue was driven by the airplane flight director. 
Glideslope and localizer raw data indicators which included a deviation scale and angular deviation 
indicators were provided (i.e., glideslope and localizer deviation). Radar altitude was shown digitally 
underneath the flight path marker when below 2500 ft above ground level (AGL). In addition, a 
HUD flare cue, consisting of a flare “prompt”, was provided where, at 50 ft AFL, two “plus” signs 
flashed above the flight path marker (Figure 4). 



Figure 2. EFVS HUD (left) and SYS HUD (right) 


14 




j Flight Path Marker & 

Wind Speed & Direction \ Guidance Cue 



Figure 3. HUD Symbology 



Figure 4. HUD Flare Cue 


The PF had independent controls to adjust the stroke symbology brightness and the raster imagery 
brightness and contrast. The pilots were trained on how to set the brightness/contrast of the SVS 
image. They were allowed to adjust it at any time during the test, and the principal investigator 
specifically had them set it at the beginning of the day to their personal preferences. The PF also had 
a declutter control, implemented as a four-button castle switch on the pilot’s sidestick. The four 
“declutter” states available to the PF were: (1) Declutter All (no symbology or imagery); (2) 
Symbology (Stroke) Toggle on/off; (3) Imagery (Raster) Toggle on/off; and (4) Display All (both 
symbology and imagery). 

The HUD was stowed when not being used to avoid any confounding from the HUD being in place 
during “non-HUD” runs. 
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3. 2. 5 Eye Tracking System 


A 4-camera Smart Eye remote eye tracking system was installed (Figure 5) and optimized to at a 
minimum track the left seat pilot head position with six degrees of freedom at all times. 

The SmartEye™ remote eye tracking systems first determine head position in all six degrees of freedom. 
This is done by two dimensional image recognition using several key facial characteristics. Points such 
as the eye comers, nostrils, comers of the mouth, ears, etc. are identified and measured in relative pixel 
distance. Combining the located image points using two cameras of known position allows for 3D image 
processing, producing six degree of freedom head position values. Eye tracking is then measured by 
determining the center of the pupil through contrast image processing, relative to a glint reflection, 
provided by infra-red light sources of known location on the iris that indicates the center of the eye itself. 
By calculating the known distance between these two points, trigonometry is used to calculate a vector 
between the two points. A three dimensional eye gaze vector can be calculated in reference to a world 
coordinate system, such as a flight deck. A minimum of two cameras are required to perform three 
dimensional calculations. 

Eye gaze vector tracking was optimized for the HDD instmments and OTW in the pilots’ forward looking 
field of view. The system was not optimized for accurate lateral tracking beyond the ND or to the left of 
the wing panel on the glare shield. 



Figure 5. RFD Smart Eye Camera and Flasher Locations 


17 




3.3 SV Simulations 


A SV database was developed by NASA starting from the OTW database, generally following the 
standards from RTCA DO-3 15B [22]. The database used a one arc-second Digital Elevation Model 
(DEM) of a 1 10.25 nautical miles (nm) (East-West) by 145.6 (North-South) nm area centered around 
ORD. The DEM was draped with an elevation-based coloration texturing. 

Each ORD runway was modeled as an asphalt-colored polygon using the threshold data and runway 
widths. Threshold lines, edge lines, and runway numbers were added. 

The intended landing runway (as selected through the FMS prior to run initiation) was denoted on the 
primary display concept being evaluated, either as a conformal magenta outline on the head-down 
PFD depiction, or an 8000 x 200 ft outline (shown as edge lines) on the HUD. 

Because the test was confined to low altitude approach, landing, and surface operations at ORD, 
obstacles would not create a significant visual cue. Therefore, obstacles were not included or marked 
in the SV depiction. 

The SVS-PFD symbology mirrored the HUD using conformal depictions for the flight path marker, 
single cue flight path-referenced guidance symbology, and flight path angle reference cue. Other 
required primary flight reference information was also drawn (e.g., airspeed, altitude, and raw data 
deviations). 

When drawn on the HUD, the SV database terrain texturing and coloration was slightly changed to 
improve its visual perception primarily by specific coloration for conversion into a gray-scale format. 

The SV depiction was always drawn in a heading-up format. Any crosswind was evident by 
conformal lateral positioning of the flight path marker. However, the flight path marker and guidance 
cue were limited and displayed as ghosted representations if their conformal positions exceeded pre- 
determined values. 

3.4 EV Simulation 

The EV real-time simulation is created by the Evans and Sutherland EPX™ physics-based sensor 
simulation. EPX provides rendering of airports, complex terrain, advanced weather, and other high- 
resolution three-dimensional effects for flight simulation. The ORD database was instantiated with 
material code properties. From this database, an IR sensor simulation, interacting with this material- 
coded database and the simulated weather conditions, created the desired test experimental 
conditions. 

The EV simulation mimicked the performance of a short- wave/ mid- wave FLIR, using an 
approximately 1.0 to 5.0 micron wavelength detector. The nominal enhanced visibility was 
approximately 2400 ft for this experiment. 

The eye point reference for the EV simulation was placed five ft below the pilot design eye reference 
point, but otherwise properly boresighted (i.e., angular alignment) to the aircraft. In the simulated 
airplane, the pilot is approximately 20 ft above the ground during surface operations. This EV eye 
point reference/parallax error generates 2.5 milliradian error to a point located 2000 ft away - 
approximately half of the accuracy budget of the EFVS per current RTCA DO-315 accuracy 
requirements [16]. 
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3.5 Navigational Performance Variations 

Variations in navigational accuracy were simulated on each run, bounding ±12 ft vertical and ±12 ft 
horizontal deviations from the true position. These values were determined by using measured 
performance data found in the Global Positioning System Wide Area Augmentation System (WAAS) 
performance standard document [25]. This effect was added for realism in positioning system 
accuracy. The selected inaccuracies were randomly varied across each subject’s test matrix and were 
held constant during a run. 

3.6 Automatic Dependent Surveillance - Broadcast 

Expected Automatic Dependent Surveillance-Broadcast (ADS-B) inaccuracies were simulated [26]. 
Airborne traffic position and velocity data included Gaussian position and velocity errors about their 
true values representative of RTCA DO-289, Navigation Accuracy Category for Position (NACp) = 9 
(i.e., 95% accuracy bound on horizontal position of 30 m) and Navigation Accuracy Category for 
Velocity (NACv) = 2 values (i.e., 95% accuracy bound on horizontal velocity of 3 m/sec). Surface 
traffic (i.e., aircraft with altitudes less than 100 ft height above threshold, or HAT) included Gaussian 
position and velocity errors about their true values representative of RTCA DO-289, NACp =11 
values (i.e., 95% accuracy bound on horizontal position of 3 m) and NACv = 4 values (i.e., 95% 
accuracy bound on horizontal velocity of 0.3 m/sec). Traffic data were updated at a one hertz rate to 
emulate ADS-B transmission rates. Between updates, the traffic position data were estimated by 
first-order inter-sample projection of the one hertz data. An ADS-B latency of 0.6 seconds was also 
emulated. 

3.7 Crew Display Concepts 

Two head-down flight display concepts and three head-up flight display concepts were evaluated by 
the crews while flying approaches, landings, and surface operations to Runways 4R, 9R, 22L or 22R 
at ORD. 

3. 7 . 1 Head-Down Flight Display Concepts 

The two HDD concepts (referred to as the Conventional HDD and SVS HDD) are shown in Figures 6 
and 7, differing from each other only in the absence or presence of SVS on the PFD. The HUD was 
stowed during HDD evaluations. 

The SVS on the PFD portrayed a 33° V x 44° H field-of-regard. Assuming a 25-inch distance from 
the Design Eye Reference Point to the display, the SVS concept had a minification factor of 
approximately 2.1 for the PF. The PF left display also had a datalink message area and Horizontal 
Situation Indicator (HSI). The PM right display showed a quad-view of flight information: a PFD 
(upper left); HSI (lower left); FFIR repeater or blank area (upper right); and datalink message area 
(lower right). The upper right area of the quad-display was blank for conventional and SVS HDD 
evaluations. 

The PF (right display) and PM (left display) NDs always showed flight traffic and navigational 
information in the airborne mode (see Airborne Modes on Figure 6). The PF and PM NDs 
transitioned to a moving map mode when on the ground and groundspeed less than 100 knots (see 
Surface Modes on Figure 7). The PM ND included a runway inset view in both airborne and moving 
map modes. 
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The surface moving map and the runway inset views always showed their respective images of the 
airport and intended landing runway, but traffic icons (i.e. CDTI) were only drawn when being 
experimentally evaluated. The CDTI configurations (none, on Moving Map, on Moving Map and 
Runway Inset, see Figure 8) were balanced across the experiment matrix. These experimental 
variations (e.g., the presence and absence of CDTI) were conducted to test the time required, accurate 
identification, and pilot workload associated with recognizing and reacting to potential ground 
collisions or conflicts with other aircraft. 


Airborne Modes 


Conventional Displays 



PF Left Display PF Right Display 



SVS Displays 



Figure 6. Airborne Head-Down Display (HDD) Concepts: Conventional HDD on top and SVS HDD on bottom 
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Surface Modes 



PF Left Display 



SVS Displays 


PF Right Display 




Figure 7. Surface Head-Down Display (HDD) Concepts: Conventional HDD on top and SVS HDD on bottom 





Figure 8. CDTI formats - None (left), Moving Map (center), and Moving Map/Runway Inset (right) 


3. 7.2 Head-Up Flight Display Concepts 

The three HUD concepts (referred to as the Conventional HUD, SVS HUD and EFVS) were tested, 
differing from each other only in the absence or presence of imagery (SV or FLIR) on the PF HUD. 

The Conventional HDD (i.e., no SVS) concept was displayed during these HUD runs. The PM 
(head-down) right display showed FLIR imagery in the upper right corner of the quad- view for EFVS 
runs and was blank (black in color) for Conventional and SVS HUD runs. Figure 2 (in the left-side 
view) shows the EFVS display concept with symbology and FLIR imagery. 

3.8 Simulator Recording 

Engineering unit data and audio and video data were recorded. All data were time-stamped for 
correlation and subsequent post-test analysis. 
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In addition, a SmartEye™ head and eye-tracking system was installed and used to measure, as a 
minimum, the pilot’s head-position to infer where s/he was looking. Pilot visual behavior was 
quantitatively calculated based upon three-dimensional gaze vector and head position analysis to 
evaluate variation in pilot behavior. 

Video recordings were made of a combination of the following: 

• Pilot and Co-Pilot’s PFD and ND 

• Cockpit area camera; OTW image 

• Smart Eye™ Eyetracking Video Data 

• HUD Camera 

3.9 Evaluation Task 

The evaluation task was a straight-in Instrument Landing System (ILS) approach with a 3 -degree 
descent angle to one of four ORD runways (4R, 9R, 22R, or 22L). The approach started three nm 
from the runway threshold for each run. The weather consisted of low to moderate winds with either 
10 knot headwind, 10 knot tailwind, 7.5 knot crosswind, or 15 knot crosswind, light turbulence (root- 
mean-square (RMS) of 1 ft/sec), and varying OTW visibility levels (1800 ft, 1400 ft, or 1000 ft 
RVR). When used experimentally, the enhanced flight visibility (i.e., visibility provided by the FLIR 
sensor) was 2400 ft RVR. The PF hand-flew the approach from the left seat with the auto-throttle set 
to “speed-hold” at the approach speed of 130 knots indicated airspeed. The auto-throttle 
automatically reduced to idle thrust at 35 ft AGL for landing. The run was terminated once the PF 
completed the landing, roll-out and turnoff or upon go-around initiation. The aircraft was configured 
to land prior to each run (landing gear down and flaps 30 degrees). 

The PFs were instructed to fly the aircraft as if there were passengers aboard, track the approach path, 
and land within the touchdown zone with an acceptable sink rate. After landing, they were to capture 
the centerline and exit at the expected taxiway at a speed of 5 to 15 knots at the 90 degree exits or 30 
knots at the high-speed exits. They were also instructed to initiate a go-around if the landing was not 
safe or there were any safety concerns during the approach. 

3.10 Crew Procedures 

The crew was trained in monitored approach crew procedures. The PF flew the approach using the 
HUD or HDD as the primary flight reference. The PM monitored using the available HDD 
information, including a FLIR repeater (when EFVS was flown), and the OTW scene and assisted the 
PF as appropriate and necessary. There was no transfer of control from the PF to PM (or vice versa). 

To facilitate comparison between baseline conditions (i.e., no SVS or EFVS installed) and 
SVS/EFVS configurations, crew procedures were standardized and trained. 

3.10.1 Baseline and SVS Procedures 

The procedures for the baseline (i.e., no SVS or EFVS installed) and SVS configurations were 
identical and followed normal crew instrument approach procedures (see Table 1). One of the 
intended functions of the SVS is to improve the pilot’s ability to conduct the instrument portion of the 
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approach - not to enable descent below the published DA/DH (decision altitude/decision height). 
Therefore, no change in crew procedures would be expected. 

With SVS equipage, the crew (especially, the PM) must monitor the validity of the SVS to continue 
the approach. Prior to reaching the published DA/DH, the crew must verify that the SVS image, 
including the guidance, flight path marker, and runway are functioning properly, without unusual 
behavior statically or dynamically. 

The ability to descend below the DA/DH and to descend below 100 ft HAT was dependent upon the 
pilot being able to recognize and identify, using natural vision, the required visual references, shown 
in the left hand column of Table 2. 


Table 1. Baseline and SVS Crew Procedures 


Altitude- 
Based Events 

Radio/Baro 
Altitudes (ft) 

Automatic 

Callouts 

PF 

Tasks/Callouts 

PM 

Tasks/Callouts 

500 ft 

500/1196 

“500” 

Response: 

“Check” 


Minimums 
+ 100 ft 

DA/DH+100 

“Approaching 
Minimums ” 

Response: 

“Roger” 


Published 

DA/DH 

“ Minimums ” 

With Approach Lights, 

With "Out the Window" Cues, 

Minimums 



Call “Approach Lights , Down to 100” 

Call “Lights” or “Field in Sight” 




Without Approach or Landing 
References, 

Call “Going Around” 

Without PF Call of 'Approach 
Lights', 

Call “Go Around” 

100 ft HAT 

100 / 796 

“100” 

With "Out the Window" Landing 

With "Out the Window" Cues, 




References, 

Call “Field in Sight , Landing” 

Call “Lights” or “Field in Sight” 




Without "Out the Window" Landing 

Without PF Call of 'Landing', 




References, 

Call “Going Around” 

Call “Go Around” 
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Table 2. Required Visual References 


Required Visual References Using 
Natural Vision 
(14 CFR § 91.175 (c)) 

Required Visual References Using an 
Enhanced Flight Vision System 
(14 CFR §91.175 (1)) 

For operation below DA/DH or MDA - 

At least one of the following visual references for the intended 
runway must be distinctly visible and identifiable: 

Approach light system 
Threshold 
Threshold markings 
Threshold lights 
Runway end identifier lights 
Visual approach slope indicator 
Touchdown zone 
Touchdown zone markings 
Touchdown zone lights 
Runway 

Runway markings 
Runway lights 

For operation below DA/DH or MDA - 

The following visual references for the intended runway 
must be distinctly visible and identifiable: 

Approach light system 

OR 

Visual references in BOTH paragraphs 
91.175(l)(3)(ii)(A) and (B) — 

(l)(3)(ii)(A) The runway threshold, identified by at 
least one of the following - 

-- beginning of the runway landing surface, 

-- threshold lights, or 
-- runway end identifier lights 

AND 

(l)(3)(ii)(B) The touchdown zone, identified by at 
least one of the following - 

-- runway touchdown zone landing surface, 

-- touchdown zone lights, 

-- touchdown zone markings, or 
-- runway lights. 



Descent below 100 feet height above TDZE - 

At least one of the following visual references for the intended 
runway must be distinctly visible and identifiable: 

Approach light system, as long as the red terminating bars or 
red side row bars are also distinctly visible and identifiable 
Threshold 
Threshold markings 
Threshold lights 
Runway end identifier lights 
Visual approach slope indicator 
Touchdown zone 
Touchdown zone markings 
Touchdown zone lights 
Runway 

Runway markings 
Runway lights 

Descent below 100 feet height above TDZE - 

The following visual references for the intended runway 
must be distinctly visible and identifiable: 

The lights or markings of the threshold 

OR 

The lights or markings of the touchdown zone 


3.10.2 EFVS Procedures 

The EFVS procedures used for this study (see Table 3) were built around common practice in current 
EFVS operations and FAA requirements (14 CFR § 91.175 (1)) but extended to emphasize that to 
descend below the DA/DH and to descend below 100 ft height above the TDZE depended upon the 
PF being able to recognize and identify the required visual references using EFVS , as shown in the 
right hand column of Table 2. For this test, the crews were briefed that the enhanced flight visibility 
was not less than the visibility prescribed by the instrument approach procedure being used. 
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Table 3. EFVS Crew Procedures 


Altitude-Based 

Events 

Radar/Baro 
Altitudes (ft) 

Automatic 

Callouts 

PF 

Tasks/Callouts 

PM 

Tasks/Callouts 

500 ft 

500/1196 

“500" 

Response: 

“Systems Normal EVS Normal " 


Minimums 
+ 100 ft 

DA/DH+100 

“Approaching 
Minimums " 

Response: 

“Roger" 


Published 

Minimums 

DA/DH 

“Minimums" 

With EFVS Approach Lights, 

With "Out the Window" Cues, 

Call “EVS Lights" 

Call “Lights" or “Field in Sight" 

Without EFVS Approach Lights, 
Call “Going Around" 

Without PF Call of TVS Lights', 
Call “Go Around" 

EFVS Decision 
Altitude 
(100 ft AFL) 

100 / 796 

“100" 

With EFVS Landing Visual References, 
Call “Landing" 

With "Out the Window" Cues, 
Call “Lights" or “Field in Sight" 

Without EFVS Landing Visual 
References, 

Call “Going Around" 

Without PF Call of 'Landing', 
Call “Go Around" 


3.11 Experiment Matrix 

Both single pilot and two-crew evaluations were conducted, but the primary emphasis was placed on 
the two-crew evaluations. 

The primary two-crew experiment test matrix aligned three visibility conditions against five SEVS 
configurations (combinations of displays and vision systems) as shown in Table 4. The visibility 
conditions and SEVS configurations evaluated are indicated by an ‘x’ in Table 4. Each of these ‘x’- 
marked conditions was flown twice, once with TDZ/CL lights and once without TDZ/CL lights. 

Two baseline conditions are identified in Table 4: one using a HUD and one using HDDs; both 
without any SVS or EFVS information. These baseline conditions were defined by the FAA for this 
experiment to represent operational concept baselines with which to compare the four SVS 
operational concepts (HUD in 1000 and 1400 ft RVR and HDD in 1400 and 1800 ft RVR) and two 
EFVS operational concepts (1000 and 1400 ft RVR) tested. 

The baseline comparison for SVS operational concepts is the conventional HUD flown in 1400 ft 
RVR without TDZ/CL lights. This configuration mimics the current Special Authorization Category 
I approach specified in FAA Order 8400.1 3D. This authorization does not require the use of 
centerline or touchdown zone lights. This configuration will be used for comparative analysis in two 
primary performance aspects: 1) the FTE to the DH and at the DH; and, 2) the pilot’s ability to 
transition from instrument flight to visual flight, identify the required visual references, and acquire 
the runway environment. In order for a HDD SVS configuration to be considered for the same 
operational approval as the HUD on a Special Authorization Category I approach, the FTE, pilot 
performance, and touchdown and rollout performance using the HDD SVS configuration should be 
no worse than the performance achieved from an authorized Special Authorization Category I 
operations. The pilot’s ability to transition to head-up visual flight must be smooth and seamless. 


The baseline comparison for EFVS operational concepts is the conventional HDD with 1800 ft RVR 
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with TDZ/CL lights. This configuration mimics the lowest visibility currently authorized for certain 
Category I approaches without using a HUD (see FAA Order 8400.1 3D). This configuration will be 
used for comparative analysis in two primary performance aspects: 1) the ability of the pilot to fly 
below the DA/DH to the runway; and, 2) the pilot’s ability to safely land in the touchdown zone, with 
an acceptable sink rate, and to maintain tracking and alignment with the runway centerline upon 
landing roll-out. For the baseline condition, below the DA/DH, the pilot’s primary reference for 
maneuvering the aircraft will be the OTW visual cues (i.e., natural vision). In the case of an EFVS, 
the pilot’s primary reference for maneuvering the aircraft will be the enhanced flight vision cues. 

For a HUD EFVS configuration to be considered for operational approval to use EFVS in lieu of 
natural vision to descend below the DA/DH and land and roll-out, equivalent performance to an 
approach where natural vision is relied on below the DA/DH must be demonstrated. 


Table 4. Experiment Matrix for Two-Crew Operations 



Visibility 

Display 

Vision 

System 

1800 ft RVR 

1400 ft RVR 

1000 ft RVR 

HUD 

None 


X 



SVS 


X 

X 


EFVS 


X 

X 

HDD 

None 

X* 

X 



SVS 

X 

X 



* 200 ft DA; All others 150 ft DH 


EFVS and SVS non-normal runs were injected into the test unbeknownst to the PF and PM to assess 
the crew’s decision-making process when confronted with these non-normal events while flying in 
low-visibility conditions. The crews did not receive any briefing or training on these non-normal 
runs. The non-normal runs included runway incursion scenarios, SVS lateral and vertical navigation 
system error scenarios, EFVS HUD failure scenarios, and for EFVS, insufficient enhanced flight 
visibility to land scenarios. 

To accomplish the runway incursion testing, the experiment was conducted in two testing phases (a 
“non-expectancy” phase and an “expectancy” phase) which were separated by the staging of a 
runway incursion, serving as a rare event collision scenario. In the first phase, experimental 
variations in EFVS and SV equipage and varying weather/visibility conditions were conducted (see 
Table 4), and these runs were considered normal runs. Randomly placed within the normal run 
matrix of the first phase were five runs with non-normal events: two runs with SVS navigation system 
error (one lateral, lone vertical), two runs with EFVS HUD failure (one in 700 ft RVR and one in 
1000 ft RVR), and one run with insufficient enhanced flight visibility to land. 

The two SVS non-normal events involved unannunciated navigation system inaccuracies while flying 
the SVS HDD. These deviations were: a) lateral deviation of +/-131 ft (left/right) or b) vertical 
deviation of +/-1 15 ft (high/low). The lateral and vertical inaccuracies chosen are the WAAS 
horizontal and vertical alert limits, respectively. The lateral navigation system error was always 
flown to runway 22R without TDZ/CL lights in a 10 knot headwind and 1400 ft RVR OTW test 
condition. The vertical navigation system error run was always flown to runway 22L without 
TDZ/CL lights in a 10 knot headwind and 1400 ft RVR OTW test condition. These configurations 
were used to assess the flight crew’s reaction to a situation where the navigation positioning system 
provides an unannunciated large error. 
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The two EFVS non-normal events gauged the crew’s reaction and consequences of a failure of the 
EFVS at 50 ft AFL. The nominal EFVS (FFIR) visibility was 2400 ft. The OTW weather and 
visibility were either 1000 ft RVR or 700 ft RVR when a failure of the HUD caused the loss of all 
HUD information (i.e., loss of HUD symbology and EFVS) at 50 ft AFF. These two non-normal 
runs were flown without TDZ/CF lights. The 1000 ft RVR EFVS failure runs were flown in a 7.5 
knot left crosswind and the 700 ft RVR EFVS failure runs were flown in a 7.5 knot right crosswind. 
The criticality of this failure is that the pilot loses both the enhanced vision view of which s/he is 
reliant and any guidance information, causing the pilot to rely solely on the available OTW visual 
cues to complete the landing and roll-out or go-around. The crews were not briefed or trained on this 
failure event. 

An additional EFVS run with the enhanced flight visibility set to 1000 ft (instead of the nominal 2400 
ft) was flown during the first phase of testing. For this HUD EFVS run, the OTW visibility was set to 
1000 ft RVR and flown to a runway without TDZ/CF lights. This run was added to evaluate the 
tendency of the PF to continue an approach to landing even though s/he did not necessarily have the 
enhanced flight visibility sufficient to conduct the operation. The crews were not briefed that the 
enhanced flight visibility had been reduced to 1000 ft. 

The last run of the first phase was a “rare event” runway incursion scenario which was flown to test 
pilot/crew recognition and reaction in a non-normal situation without expectancy of the flight crew. 
The “unexpected” runway incursion was flown using one of four display configurations: 1) EFVS 
without CDTI, 2) EFVS with CDTI on moving map and runway inset, 3) SVS HDD without CDTI, 
or 4) SVS HDD with CDTI on moving map and runway inset. So, there were 12 total unexpected 
runway incursion runs for the simulation experiment, with three samples in each of the four display 
configurations tested. This run was always flown to runway 22R without TDZ/CF light in a 10 knot 
headwind and 1400 ft RVR OTW test condition. Because of the severity of the rare event runway 
incursion, it was anticipated that the subsequent behavior of the crew (pilots) would be altered and 
more attuned to potential traffic incursion events. Hence, this run was always the final one in the first 
phase testing. 

The second phase followed the rare event runway incursion scenarios. In this second phase, repeated 
incursions/object detection scenarios were flown and tested using EFVS and one of three CDTI 
combinations (none, Moving Map, Moving Map and Runway Inset). Each crew was exposed to nine 
EFVS CDTI evaluations of “expected” traffic incursion on either a runway or taxiway: three 
incursion events occurred on runways without TDZ/CF lights (4R, 22F, and 22R) and six taxiway 
incursion events. It was assumed that these runs would not be without expectancy on the part of the 
pilots. The purpose of the second phase runs was to test the effects of CDTI on the time required, 
accuracy of identification, and pilot workload associated with potential ground collisions or conflicts 
with other aircraft. 

The single pilot conditions (see Table 5) focused primarily on collecting data for correlation or 
comparison with planned follow-on flight test configurations and for comparison of the influence of 
two-crew operations. As such, the comparisons between the two-crew and single pilot operations 
contained within this test were valid and indicative of the influence of crewed operations. These 
results will also provide data for comparison to the planned (single pilot) flight test. These runs were 
not conducted to advocate nor imply the possible acceptance of single pilot operations for Part 25 
aircraft. In addition, the general applicability of the single pilot results with respect to operations and 
equipage may not be representative of Part 23 -type aircraft. 

Only the subject trained as PF flew the single pilot evaluations. The six single pilot runs (see Table 
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5) were blocked together and were conducted within the first phase of experimental testing. TDZ/CL 
lights were present for each of these six runs. 


Table 5. Experiment Matrix for Single Pilot Operations 


Operational Concept 

CDTI 

None 

Moving Map 
Only 

Moving Map 
and Runway 
Inset 

EFVS HUD in 1000 ft RVR 

X 

X 

X 

SVS HDD in 1400 ft RVR 

X 

X 

X 


Wind variations were balanced across the experiment matrix for each crew/pilot to evenly distribute 
the conditions across the configurations. Thus, wind effects were tested but not in a within-subjects 
design. It was assumed that left and right crosswinds could be interchanged without affecting any 
experimental results. 

3.12 Measures 

During each approach and landing run, path error, pilot control inputs, and touchdown performance 
(sink rate and speed at touchdown, distance fore or aft of touchdown zone, and distance left or right 
of centerline) were measured for analysis. During taxi operations, centerline tracking was measured. 

After each run, pilots completed the Air Force Flight Test Center (AFFTC) Workload Estimate Scale 
[27]. After data collection was completed, pilots were administered two paired comparison tests: the 
Situation Awareness - Subjective Workload Dominance (SA-SWORD) [28] technique and one on 
Traffic Awareness evaluating CDTI formats tested. These subjective measures are provided in 
Appendix A. 

At six times during the two-crew testing, a traffic awareness probe, modeled after a Situation 
Awareness Global Assessment Test (SAGAT) [29] was administered. The data were used to quantify 
the flight crew’s awareness (PF and PM) of traffic and the influence of CDTI. 

On these runs, twelve aircraft were located in close proximity to and on the intersecting and adjacent 
taxiways of the active runway, along the entire length. Immediately after clearing the runway, the 
displays and OTW scene were blanked. The pilots were given a paper diagram of the active runway 
(see Appendix A, Section 8.3), including the intersecting and adjacent runways and were asked to 
recall the location of all traffic in proximity to the runway they just used. They identified this by 
circling the location of the aircraft (i.e., traffic) that they recalled on the paper chart. The pilots were 
asked to complete this probe without consulting or discussing with each other. 

The probe was administered for the following runs: 

1 . EFVS in 1400 ft RVR flown to a runway without TDZ/CE lights 

2. EFVS in 1400 ft RVR flown to a runway with TDZ/CL lights 

3. Conventional PFD in 1800 ft RVR flown to a runway with TDZ/CL lights (i.e., Operational 
Baseline for EFVS concept comparisons) 
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4. SVS PFD in 1400 ft RVR flown to a runway without TDZ/CL lights 

5. SVS PFD in 1400 ft RVR flown to a runway with TDZ/CL lights 

6. Conventional HUD in 1400 ft RVR flown to a runway without TDZ/CL lights (i.e., 
Operational Baseline for SVS concept comparisons) 

CDTI (None, On Moving Map, or On Moving Map and Runway Inset) were balanced across the test 
matrix and order of occurrence for the traffic probe. 

3.13 Test Conduct 

The subjects were given a one-hour briefing describing the experiment, HUD and HDD concepts, 
crew procedures, and evaluation tasks. The test purpose was described to the test subjects as 
“evaluating the potential use of EFVS and SVS for reduced landing weather minima and the influence 
of CDTI for NextGen operations.” 

After the briefing, a 1.5 hour training session in the RFD was conducted to familiarize the subjects 
with the aircraft handling qualities, display symbologies, pilot procedures, and controls. In particular, 
in-simulator training highlighted the crew procedures for EFVS and SVS operations and landing 
performance. Landing performance was planned as one of the performance parameters used to assess 
the efficacy of the SEVS experimental variations. However, none of the pilots were familiar with the 
handling characteristics of the RFD simulator. To accommodate this disparity, each PF was trained to 
an acceptable standard of approach and landing performance. 

In Table 6, touchdown performance criteria are shown. After each training run, a landing 
performance assessment was displayed for feedback (Figure 9). The value and rating for the 
touchdown performance assessment were color-coded. Touchdown performance parameters were 
depicted in green text if they met the “Desired” criteria, yellow text if they met the “Adequate” 
criteria, and red text if the met the “Not Adequate” criteria listed in Table 6. The pilots were asked to 
meet the desired performance criteria listed in Table 6. Training concluded once the pilots 
demonstrated repeatable desired landing performance, with only an occasional adequate performance 
score. If the adequate performance criteria were met, they landed within the touchdown zone with 
acceptable sink rates. 

The training was flown in varying OTW visibility from visual conditions down to 1000 ft RVR. 
Similarly, enhanced flight visibility (i.e., visibility provided by the FLIR sensor) ranged from 
unlimited down to 1000 ft. The training emphasized that they must always remain safe and if they 
felt unsafe conditions exist, the necessary precautions, including a go-around, should be executed 
immediately. 
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Table 6. Touchdown Performance Criteria 


Performance Value 

Desired 

Adequate 

Not Adequate 

Lateral Distance from 
Centerline 

Within +/- 27 ft 

Between +27 and +58 ft or 
Between -27 and -58 ft 

> +/-5S ft 

Longitudinal Distance from 
Threshold 

Between 750 to 2250 ft 

Between 200 & 750 ft or 
Between 2250 & 2700 ft 

< 200 or >2700 ft 

Sink rate 

Between 0 to 6 ft/sec 

Between 6 to 10 ft/sec 

| >10 ft/sec 

Airspeed (knots) 

Between V re f-5 to V re f+5 

Between V re f-5 to V re f-15 

< V re f-15 or > V r ef+5 


V ref + 5 is the approach speed in table above 



Figure 9. Landing performance assessment 


4 Results 

Flight performance (approach, landing, and rollout), pilot workload, and eye-tracking (where 
applicable) data were analyzed for two operational domains: EFVS Operations and SVS Operations. 

In the first domain (referred to as EFVS Operational Concept Comparisons in Section 4.2), the 
operational feasibility, pilot workload, and pilot acceptability of conducting a straight-in instrument 
approach procedure with published vertical guidance using EFVS (i.e., FLIR imagery on a HUD) for 
the approach, landing, roll-out and turn-off in weather and visibility as low as 1000 ft RVR were 
evaluated. The baseline comparison for EFVS operational concepts is the Conventional HDD flown 
in 1800 ft RVR with TDZ/CL lights. This baseline condition creates the direct comparison of “visual 
segment” performance (from DA/DH to touchdown and roll-out) in the lowest visibility (1800 ft 
RVR) allowable using natural vision under today’s regulations against an EFVS “visual segment.” 

Similarly, in the second domain (referred to as SVS Operational Concept Comparisons in Section 
4.3), the operational feasibility, pilot workload, and pilot acceptability of conducting a straight-in 
instrument approach to a 150 ft DA/DH procedure with published vertical guidance using SVS 
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(displayed either head-down or head-up) and to transition to natural OTW visual conditions for 
landing in weather and visibility as low as 1400 ft RVR were evaluated. The baseline comparison for 
SVS operational concepts is the Conventional HUD flown in 1400 ft RVR without TDZ/CL lights. 
This baseline condition creates the direct comparison of “instrument segment and visual transition” 
performance (descent to the DA/DH with transition to the visual segment) in the lowest visibility 
(1400 ft RVR) allowable using natural vision under today’s regulations (i.e., Special Authorization 
Cat. I, under FAA Order 8400. 13D) against an SVS “instrument segment and visual transition.” 

EFVS and SVS non-normal runs were injected into the test unbeknownst to the PF and PM. These 
situations stressed the crew’s decision-making process when confronted with non-normal events and 
are analyzed separately and discussed in the next section. 

Single pilot evaluations were also conducted for the EFVS and the SVS PFD with the three variations 
of CDTI (none, Moving Map, Moving Map and Runway Inset). Statistical analyses were conducted 
on approach and touchdown performance with crew complement (single, dual) and CDTI as the main 
factors. 

4.1 Metrics 

4 . 1 . 1 Flight Performance 

Flight performance was evaluated using different metrics and measures to explore specific 
assessments of interest. 

Approach performance during the “instrument” segment was analyzed using RMS localizer deviation 
(in dots), RMS glideslope deviation (in dots), and RMS sink rate deviation (in feet per minute, or 
fpm) where this value is difference or deviation from the sink rate required to perfectly track the 
glideslope in the given wind conditions. These parameters correspond intuitively to how well a 
stabilized approach to landing - an important safety measure - was established and maintained. The 
approach data were analyzed from 1000 ft to DA/DH for the normal runs. The beginning altitude 
value of 1000 ft was the start of each run. 

Approach performance was also analyzed using existing FAA [30, 31] and Joint Aviation Regulations 
(JAR) All Weather Operations (AWO) [32] performance-based approach standards for glideslope and 
localizer tracking. These standards were drawn from numerous sources pertaining to the general 
concept of low-visibility approach and landings. However, none of these existing standards were 
written specifically as quantitative performance standards for advanced vision systems (such as SVS 
and EFVS) operations but are applied herein for comparative purposes. 

A synopsis of these existing quantitative performance requirements are shown in Table 7. In the 
Practical Test Standard [30], glideslope and localizer performance requirements are expressed in 
microamps, with 150 microamps equal to full scale deflection on the ILS. In Table 7, glideslope and 
localizer deviations are expressed in dots deflection by assuming + 1-2 dots full scale deflection 
corresponds to +/-150 microamps deviation from on-course. Note that +/- 2 dots full scale deflection 
of the glideslope deviation indicator is 1.4 degrees (+/- 0.7 degrees from center of glideslope beam) 
and that +/- 2 dots full scale deflection of the localizer deviation scale is 5.0 degrees (+/- 2.5 degrees 
from runway centerline) [33]. 

This synopsis emphasizes “performance” parameters of interest that are relevant to this experiment 
and does not include many important regulation facets and nuances for the sake of brevity. Of 
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particular note, airspeed, sink rate, and bank angle control are typically evaluated but are not reported 
herein. 


For instance, AC120-29A [31] documents an acceptable means for obtaining approval of operations 
in Category I/II landing weather minima (i.e., a DH lower than 200 ft but not lower than 100 ft and a 
runway visual range not less than 1200 ft). Part of the operation includes a “visual segment using 
natural vision;” that is, below the DH, the primary reference for maneuvering the airplane is based on 
what the pilot sees visually. AC120-28D [34] describes an acceptable means for obtaining approval 
of operations in Category III landing weather minima (i.e., DHs below 100 ft, or no DH and a runway 
visual range not less than 700 ft). Unlike AC120-29A, AC120-28D does not include a natural visual 
segment. Consequently, with a goal of operating in Category III landing weather minima by use of 
SVS and EFVS technology, a conundrum is created. Further operating credit for vision systems (VS) 
technology pushes into the AC120-28D weather and visibility arena but AC120-28D does not give 
consideration for a visual segment for the pilot. The analyses that follow examine the applicability of 
existing FAA and JAR approach standards for VS operations in all-weather operations. JAR AWO- 
23 1 documents an acceptable means for obtaining approval of operations for approaches with 
decision heights below 200 ft and down to 100 ft. Like AC120-29A, the JAR standard relies on the 
pilot’s natural vision for maneuvering the airplane below the DH. 


Table 7. Quantitative approach performance standards. 



Localizer Tracking 

Glideslope Tracking 

Practical Test Standard 
Reference 30 

<3/4 Full Scale Deflection (i.e., 1.5 
dots), Between Final Approach Fix and 
Decision Height 

<3/4 Full Scale Deflection, (i.e., 1.5 dots) 
Between Final Approach Fix and Decision 
Height 

AC 120-29, 

Appendix 2, Paragraph 
6.2.1. 

Reference 31 

< 1/3 (i.e., 2/3 dots) Full Scale 
Deflection from 1000 ft Height Above 
Touchdown (HAT) to 200 ft HAT 

< 1/2 Full Scale (i.e., 1 dot) Deflection from 
700 ft HAT to 200 ft HAT 

"Cat 2, Successful 
Approach" 

FAR Part 91, Appendix A, 
Section 3, Subsection e2 

At 100 ft DH, cockpit is within and 
tracking so as to remain within, the 
lateral confines of the runway 
extended. 

Deviation from glideslope after leaving the 
outer marker does not exceed 50% Full 
Scale Deflection (i.e., 1 dot down to 100 ft 
DH) 

Joint Aviation Regulations 
-All Weather Operations 
AMC AWO 231 
Reference 32 

No more than 5% of approaches with 
>1/3 dot between 300 ft and 100 ft 
HAT 

No more than 5% of approaches with >1 
dot between 300 ft and 100 ft HAT 


Flight path performance data (lateral position, vertical position, and sink rate) were used to evaluate 
how effectively the pilots could use the different EFVS and SVS concepts during the visual segment 
of the approach to position the aircraft for landing. The absolute values for maximum bank angle 
from threshold crossing to touchdown and the bank angle at touchdown were also used to evaluate the 
effectiveness of the different EFVS and SVS concepts. 

Touchdown statistics were used to evaluate how effectively the pilots could land with the different 
SEVS display concepts and how well the crews met the touchdown performance criteria to which 
they had been trained (see Table 6). Existing FAA AC120-28D [34, Appendix 3] and JAR AWO 
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[32] performance-based “auto-land” standards for touchdown (T/D) longitudinal position, lateral 
position from centerline, and sink rate were applied in the objective landing data analysis. 
Specifically, the standards require no longitudinal touchdown earlier than a point on the runway 200 
ft from the threshold or beyond 2700 ft from the threshold, no lateral touchdown with the outboard 
landing gear more than 70 ft from the runway centerline, and no touchdown sink rate greater than -10 
feet/second to a probability of 1 x 10' 6 . These standards pertain to the general concept of low- 
visibility approach and landings using guidance systems technologies, but were not written 
specifically for operations with advanced vision systems such as EFVS and SVS. This experiment 
used an aim point located 1000 ft from runway threshold. For the simulated aircraft, the outboard 
landing gear would be 70 ft from the centerline when the fuselage is at 58 ft lateral deviation from 
centerline, assuming no crab angle at touchdown. 

Fateral deviation from centerline statistics (maximum value and RMS) were used to evaluate how 
effectively the pilots could maintain centerline during rollout with the different EFVS and SVS 
display concepts. 

4.1.2 Pilot Workload 

Workload was assessed after each experimental run, independently for the PF and PM, using the 
AFFTC Workload Estimate Technique [27]. Workload ratings were evaluated by conducting 
separate Analysis of Variance (ANOVA) tests for the EFVS operational concepts and for the SVS 
operational concepts with operational concept as the main factor. If a significant F-value was 
obtained in an ANOVA, then Student-Newman-Keuls (SNK) post-hoc tests (a set at 0.05) were 
performed. SNK post hoc tests provide specific information on which means are different from each 
other when a significant F-test result is found on a main factor consisting of three or more levels. 

4.1.3 Eye Gaze 

Several metrics were applied to the eye gaze tracking data to evaluate the PF visual behavior, 
including; head-up percentage and transition count between OTW and head-down. For HDD 
concepts, the height above threshold and pilot gaze direction at the point of visual transition OTW 
was also determined to evaluate variance in pilots’ behavior when attempting to acquire the runway 
environment. ANOVAs were conducted to determine significant variance in visual behavior across 
EFVS and SVS operational concepts. If a significant F-value was obtained in an ANOVA, then, 
Tukey Honestly Significant Difference (HSD) pair-wise comparison tests (a set at 0.05) were 
performed. HSD post hoc tests provide specific information on which means are different from each 
other when a significant F-test result is found on a main factor consisting of three or more levels. 

To further inspect visual behavior, data was divided into instrument, instrument-to- visual, visual, 
flare, and landing segments, shown below in Figure 10. Segments were chosen based upon standard 
pilot visual behavior at various heights while on short final in low visibility conditions. During each 
segment under manual flight control and auto-throttles, the pilot visual behavior is driven by known 
task loading. The instrument segment, from the initial start of 1000 ft to 50 ft above the DA/DH, 
requires the pilot to maintain attention on the primary flight display, maintaining flight on course and 
glideslope. Outside visual references are not available so attention is likely maintained inside the 
cockpit when only the HDDs are available. The instrument-to-visual segment is an altitude driven 
window, 50 feet above the DA/DH to the DA/DH, during which pilots would transition from the 
HDD to OTW to acquire the required visual references to continue to land. The visual segment 
begins from the DH to 50 feet HAT, with expected task loading to drive pilot attention out the 
window to complete the visual approach. From 50 feet HAT until all-wheels were in continuous 

33 



contact represented the flare segment, during which the task of landing is predicted to drive pilot 
attention OTW to complete the flare based upon visual references. The landing segment evaluates 
pilot visual behavior from all-wheel contact until the ground speed reaches 60 knots, at which point 
the approach to land and rollout is considered complete and taxi operations begin. 


150 ft HAT 



Landing Approach 

(50ft, " Fiore * to Runway Exit) ( tAF to 50 ft, "Flare”) 

Figure 10. Eye Tracking Analysis Segments 

Analysis to determine visual transition altitude and direction were broken into two specific types of 
visual transitions: initial glance and full transition. The factors separating these transitions are 
location specific eye fixation duration and continuance of fixations in that location. Eye gaze vectors 
in the same location for a minimum of 200 milliseconds (ms) were defined as a fixation [35]. To 
evaluate when a transition occurred, data were analyzed using a moving two second gated average of 
head-up/head-do wn visual behavior to determine the difference between an initial glance OTW and a 
full transition OTW. An initial glance was determined to be the first single fixation OTW, not 
necessarily followed by sequential fixations OTW, or in other words, pilot attention concentrated 
head up for at least 200 ms at any point without continued attention OTW. A full transition occurred 
once pilot visual attention remained continuously OTW. 

Data quality was assessed post-analysis to determine the overall effectiveness of the eye tracking 
system utilized in the study. Some data dropouts did occur while in test production, resulting in an 
average data collection rate of 87% across all pilots for all analysis segments. Calibration of the 
system indicated average spatial accuracy of the eye gaze to be approximately three degrees, 
sufficient for gross measurement of head up/head down analysis. 

4. 1 . 4 Display Concept Preferences 

Crew display concept preferences for flying with in low-visibility conditions were also assessed. The 
PF and PM were independently asked to rank order the display concepts (Conventional HDD, SVS 
HDD, Conventional HUD, SVS HUD, and EFVS) from their most preferred display (rank=l) to least 
preferred display (rank=5) for flying with in low-visibility conditions. Friedman’s ANOVA [36] was 
used to test for differences among the display concepts for the pilot display preference rankings. If a 
significant difference was found, Wilcoxon non-parametric post hoc tests were used. For the 
Wilcoxon post hoc tests, a Bonferroni correction was applied and all effects are reported at a 0.01 
level of significance [36]. 
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4. 1. 5 CDTI Influences on Situation Awareness and Traffic Awareness 


ANOVAs on paired-comparisons technique responses were employed to assess Situation Awareness 
and Traffic Awareness differences for the three CDTI formats tested. If a significant F-value was 
obtained, Student-Newman-Keuls (SNK) post-hoc tests (a set at 0.05) were performed. 

4.2 EFVS Operational Concept Comparisons 

4. 2 . 1 Approach Performance - Instrument Segment 

The three concepts used for EFVS operational concept comparisons for approach performance during 
the instrument segment (from 1000 ft to DH) were: 

1) Conventional HDD with TDZ/CL lights (1800 ft RVR/200 ft DH) 

2) EFVS (1400 ft RVR/150 ft DH) 

3) EFVS (1000 ft RVR/150 ft DH) 

The Conventional HDD flown in 1800 ft RVR to a 200 ft DH and a runway with TDZ/CL lights was 
considered the operational baseline concept for the EFVS operational concept comparisons. 
Throughout the remainder of this report, it will be referred to as one of the EFVS operational 
concepts tested in this experiment. 

The EFVS display runs evaluated for the 1000 ft and 1400 ft RVR conditions included the runs made 
with and without TDZ/CL lights. In the visibility conditions tested, the TDZ/CL lights were not 
visible to the crew during the instrument segment of the approach. 

Approach Statistics 

In Table 8, the approach statistics (mean, standard deviation, minimum value, and maximum value) 
are shown for the EFVS operational concepts, including the baseline condition. Also provided in 
Table 8 are the number of runs that resulted in a go-around and the total number of runs for the EFVS 
operational concepts tested. 
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Table 8. Approach Statistics (1000 ft to DH) for EFVS Operational Concepts 



Conventional HDD 
With TDZ/CL lights 

EFVS HUD 

1800 ft RVR/200 ft DH 

1400 ft RVR/150 ft DH 

1000 ft RVR/150 ft DH 

Number of Go-Around / Number of Runs 

2/12 

0/24 

0/24 

RMS Localizer Deviation 
(dots) 

Mean 

0.05 

0.06 

0.05 

Std Dev 

0.02 

0.05 

0.03 

Min 

0.01 

0.02 

0.01 

Max 

0.07 

0.25 

0.16 

RMS Glideslope Deviation 
(dots) 

Mean 

0.28 

0.09 

0.11 

Std Dev 

0.09 

0.05 

0.06 

Min 

0.15 

0.03 

0.05 

Max 

0.46 

0.24 

0.34 

RMS Sink Rate Deviation 
(fpm) 

Mean 

177.44 

72.77 

94.05 

Std Dev 

52.16 

50.30 

81.82 

Min 

116.23 

32.31 

41.09 

Max 

287.93 

294.17 

436.99 


All EFVS display concept approaches concluded in a landing; while, two of the EFVS operational 
baseline concept (Conventional HDD with TDZ/CL Lights in 1 800 ft RVR/200 ft DH) approaches 
resulted in a go-around. Video review of these two baseline runs indicated that the PF did not have 
the required visual cues to continue the landing at the 200 ft DH. Therefore, the crew properly 
followed the EFVS crew procedures and performed the go-around. 

ANOVA Analyses 

ANOVA analyses revealed that there were no significant (p>0.05) differences for RMS localizer 
deviation among the three concepts used for the EFVS operational concept comparisons. However, 
RMS glideslope deviation (F(2,53)=32.243, p<0.0001) and RMS sink rate deviation (from a nominal 
3-degree glideslope value) (F(2,53)=9.321, p<0.0001) were significantly less for the EFVS display 
concepts compared to the Baseline concept (Conventional HDD with TDZ/CL Lights in 1800 ft 
RVR/200 ft DH). SNK post hoc tests revealed that there were no significant differences between the 
1400 ft and 1000 ft RVR EFVS display concepts for either the RMS glideslope deviation or RMS 
glideslope deviation measures. For RMS glideslope deviation, the EFVS concepts had a mean=0.10 
dots with a standard deviation (a) = 0.05 dots, and the Conventional HDD concept had a mean=0.28 
dots with g= 0.09 dots. Similarly for RMS sink rate deviation, the EFVS concepts had a mean=84.24 
fpm with g= 56.27 fpm, and the Conventional HDD concept had a mean=l 77.44 fpm with g= 52.16 
fpm. 

Objective Approach Standards Analysis 

Existing performance-based approach standards were also applied in the objective data analysis. 
These standards emphasize the maximum glideslope and localizer deviations, instead of RMS 
deviation as shown in Table 8. The approach data were analyzed only for those approaches that were 
flown to touchdown. 


The percentage of approaches which met the localizer and glideslope criteria of the four existing 
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approach performance standards are shown, broken down by EFVS operational concept in Table 9. 
Note that in Table 9, the JAR AWO values were based on the percentage of runs flown and did not 
include the statistical analysis to ensure a 95% bound as per JAR-AWO. To apply this additional 
constraint on the data, the Continuous Method [32] technique was used to calculate the probability of 
success, P(a), of meeting the AWO exceedance criteria (1/3 dot localizer, 1 dot glideslope) 95% of 
the time with required levels of confidence for the different EFVS operational concepts flown. The 
calculated P(a) values are shown in Table 10, broken down by EFVS operational concept. The 
influence of DA is shown in Tables 9 and 10, where, if the criteria window included “DA” (e.g., in 
PTS and JAR AWO), then the analysis used a corresponding altitude window down to 200 ft or to 
150 ft as appropriate. 


Table 9. Percentage approaches successfully meeting approach performance standards. 



Localizer 

Glideslope 


Conventional 
HDD with TDZ/CL 
lights 

EFVS HUD 

EFVS HUD 

Conventional HDD 
with TDZ/CL lights 

EFVS HUD 

EFVS HUD 


1800 ft RVR/ 
200 ft DH 

1400 ft RVR/ 
150 ft DH 

1000 ft RVR/ 
150 ft DH 

1800 ft RVR/ 
200 ft DH 

1400 ft RVR/ 
150 ft DH 

1000 ft RVR/ 
150 ft DH 

FAA-S-8081-4D 

100% 

100% 

100% 

100% 

100% 

100% 

AC 120-29 

100% 

100% 

100% 

90% 

100% 

95% 

FAR 91 

100% 

100% 

100% 

40% 

100% 

91% 

JAR-AWO 

100% 

100% 

100% 

40% 

100% 

95% 


Table 10. Probability of Success in meeting AWO exceedance critieria 


Display Concept 

RVR/DH (ft) 

Localizer P(a) 

Glideslope P(a) 

Number of Runs 

Number of Go-Around 

Conventional HDD 

1800/200 

100 

29 

12 

2 

EFVS 

1000/150 

100 

95 

22 

0 

EFVS 

1400/150 

100 

97 

24 

0 


The data indicate that: 

• Localizer tracking was never an issue, irrespective of the criteria. 

• Glideslope tracking performance appears to be affected by display location (head-up versus 
head-down). 

• EFVS display concepts were able to meet the JAR-AWO localizer and glideslope criteria 
from 300 ft to 100 ft HAT without any go-arounds being performed. 

4.2.2 Approach Performance - Visual Segment 

The five concepts used for EFVS operational concept comparisons for approach performance during 
the visual segment (from DH to threshold crossing) were: 

1) Conventional HDD with TDZ/CL lights (1800 ft RVR/200 ft DH) 

2) EFVS with TDZ/CL lights (1400 ft RVR/150 ft DH) 
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3) EFVS without TDZ/CL lights (1400 ft RVR/150 ft DH) 

4) EFVS with TDZ/CL lights (1000 ft RVR/150 ft DH) 

5) EFVS without TDZ/CL lights (1000 ft RVR/150 ft DH) 

The Conventional HDD flown in 1800 ft RVR to a 200 ft DH and a runway with TDZ/CL lights was 
considered the operational baseline concept for the EFVS operational concept comparisons. 

In Tables 11-13, statistics of lateral path performance, vertical path performance, and sink rate 
deviation at 100 ft HAT and at threshold crossing are provided to evaluate how effectively the pilots 
could use the different EFVS operational concepts during the visual portion of the approach segment, 
The maximum bank angle from threshold crossing to touchdown and bank angle at touchdown were 
also used to evaluate the effectiveness of the different EFVS operational concepts and are presented 
in Table 14. 


Table 11. Lateral Path Performance Statistics of EFVS Operational Concepts in the Visual Segment of the 

Approach 



Conventional 

HDD 

With 

TDZ/CL Lights 

EFVS HUD 
With 
TDZ/CL 
Lights 

EFVS HUD 
Without 
TDZ/CL Lights 

EFVS HUD 
With 

TDZ/CL Lights 

EFVS HUD 
Without 
TDZ/CL Lights 

1800 ft RVR/ 
200 ft DH 

1400 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150 ft DH 

1000 ft RVR/ 
150 ft DH 

1000 ft RVR/ 
150 ft DH 

Number of Go-Around / Number of Runs 

2/12 

0/12 

0/12 

0/10 

0/12 

Lateral Path Deviation 
at 100 ft HAT (ft) 

Mean 

- 7.7 

- 9.4 

- 4.7 

3.2 

1.7 

Std Dev 

15.7 

13.5 

14.7 

20.6 

13.4 

Max Left 

- 39.1 

- 31.5 

- 29.3 

- 33.1 

- 29.0 

Max Right 

16.8 

11.5 

17.7 

31.0 

18.2 

Lateral Path Deviation 
at Threshold Crossing 

(ft) 

Mean 

- 6.7 

- 7.9 

- 2.6 

0.3 

- 1.0 

Std Dev 

17.5 

9.7 

19.1 

16.7 

12.9 

Max Left 

- 37.2 

- 24.1 

- 36.7 

- 30.4 

- 31.4 

Max Right 

22.1 

10.3 

21.6 

26.0 

14.7 
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Table 12. Vertical Path Performance Statistics of EFVS Operational Concepts in the Visual Segment of 

the Approach 



Conventional 

HDD 

With 

TDZ/CL Lights 

EFVS HUD 

With 

TDZ/CL 

Lights 

EFVS HUD 
Without 
TDZ/CL Lights 

EFVS HUD 

With 

TDZ/CL Lights 

EFVS HUD 
Without 
TDZ/CL Lights 

1800 ft RVR/ 
200 ft DH 

1400 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150ftDH 

1000 ft RVR/ 
ISO ft DH 

1000 ft RVR/ 
150 ft DH 

Number of Go-Around / Number of Runs 

2/12 

0/12 

0/12 

0/10 

0/12 

Vertical Path 
Deviation at 100 ft 
HAT (ft) 

Mean 

2.4 

-2.6 

-5.1 

-0.8 

-4.9 

Std Dev 

18.3 

3.6 

3.5 

6.0 

4.0 

Max Below 

-27.5 

-6.8 

-9.7 

-11.1 

-11.8 

Max Above 

27.2 

3.4 

1.0 

7.5 

3.6 

Altitude at Threshold 
Crossing (ft) 

Mean 

58.6 

53.5 

50.8 

54.7 

53.3 

Std Dev 

14.2 

6.1 

00 

-p* 

oo 

7.2 

Min 

28.4 

40.3 

43.0 

47.1 

45.6 

Max 

79.4 

64.6 

68.9 

62.1 

68.4 


Table 13. Sink Rate Statistics of EFVS Operational Concepts in the Visual Segment of the Approach 




Conventional 

HDD 

With 

TDZ/CL Lights 

EFVS HUD 

With 

TDZ/CL 

Lights 

EFVS HUD 

EFVS HUD 

EFVS HUD 



Without 

TDZ/CL Lights 

With 

TDZ/CL Lights 

Without 

TDZ/CL Lights 



1800 ft RVR/ 

1400 ft RVR/ 

1400 ft RVR/ 

1000 ft RVR/ 

1000 ft RVR/ 



200 ft DH 

150 ft DH 

150 ft DH 

150 ft DH 

150 ft DH 

Number of Go-Around / Number of Runs 

2/12 

0/12 

0/12 

0/10 

0/12 

Sink Rate at 100 ft HAT 
(ft/min) 

Mean 

-688.3 

-699.4 

-704.2 

-666.4 

-689.3 

Std Dev 

165.6 

79.5 

93.2 

75.0 

72.9 

Max 

-1059.9 

-823.1 

-876.0 

-762.9 

-820.9 

Sink Rate at Threshold 
Crossing (ft/min) 

Mean 

-663.7 

-602.4 

-562.3 

-636.8 

-549.2 

Std Dev 

241.0 

130.8 

162.0 

130.9 

125.3 

Max 

-922.7 

-731.6 

-850.1 

-785.9 

-815.4 


ANOVA analyses indicated no significant differences (p>0.05) for lateral deviation from centerline, 
vertical deviation from path, or sink rate among the five EFVS operational concepts tested at both the 
100 ft HAT and Threshold Crossing Locations. 
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Table 14. Bank Angle Statistics of EFVS Operational Concepts 



Conventional 

HDD 

With 

TDZ/CL Lights 

EFVS HUD 
With 

TDZ/CL Lights 

EFVS HUD 
Without 
TDZ/CL Lights 

EFVS HUD 
With 

TDZ/CL Lights 

EFVS HUD 
Without 
TDZ/CL Lights 

1800 ft RVR/ 
200ft DH 

1400 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150ftDH 

1000 ft RVR/ 
150 ft DH 

1000 ft RVR/ 
150 ft DH 

Max Bank Angle 
from Threshold 
Crossing to 
Touchdown (deg) 

Mean 

3.5 

on 

3.4 

5.2 

4.0 

Std Dev 

2.5 

2.6 

2.4 

°o 

'St 

3.1 

Max 

9.2 

9.8 

8.8 

14.2 

11.0 

Bank Angle at 
Touchdown (deg) 

Mean 

1.9 

2.1 

1.6 

3.1 

3.3 

Std Dev 

1.9 

2.7 

0.9 

3.6 

3.3 

Max 

6.0 

9.8 

3.2 

12.4 

11.1 


ANOVA analyses indicated no significant differences (p>0.05) among the five EFVS operational 
concepts for the maximum bank angle from threshold to touchdown or for the bank angle at 
touchdown measures. 

4 . 2 . 3 Landing Performance 

The five concepts used for EFVS operational concept comparisons for landing performance were: 

1) Conventional HDD with TDZ/CL lights (1800 ft RVR/200 ft DH) 

2) EFVS with TDZ/CL lights (1400 ft RVR/150 ft DH) 

3) EFVS without TDZ/CL lights (1400 ft RVR/150 ft DH) 

4) EFVS with TDZ/CL lights (1000 ft RVR/150 ft DH) 

5) EFVS without TDZ/CL lights (1000 ft RVR/150 ft DH) 

The Conventional HDD flown in 1800 ft RVR to a 200 ft DH and a runway with TDZ/CL lights was 
considered the operational baseline concept for the EFVS operational concept comparisons. 

Landing Statistics 

In Table 15, the touchdown (T/D) statistics (mean, standard deviation, minimum value, and maximum 
value) are shown, broken out by RVR and TDZ/CL light configuration, for the EFVS operational 
concepts. The T/D measures (lateral position, longitudinal position and sink rate) means were used to 
determine which touchdown performance rating level (Desired, Adequate, or Not Adequate) as 
defined in Table 6 was achieved for the EFVS operational concepts. Also provided in Table 15 are 
the number of runs that resulted in a go-around and the total number of runs for the EFVS operational 
concepts tested. Note that for touchdown lateral position data in Table 15, the “min” value equates to 
the maximum deviation to the left of centerline and the “max” value equates to the maximum 
deviation to the right of centerline. 
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Table 15. Landing Statistics for EFVS Operational Concepts 



Conventional 

HDD 

With 

TDZ/CL Lights 

EFVS HUD 
Without 
TDZ/CL Lights 

EFVS HUD 

With 

TDZ/CL Lights 

EFVS HUD 
Without 
TDZ/CL Lights 

EFVS HUD 

With 

TDZ/CL Lights 

1800 ft RVR/ 
200ft DH 

1400 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150 ft DH 

1000 ft RVR/ 
150 ft DH 

1000 ft RVR/ 
150ftDH 

Number of 
Go-Around / 
Number of Runs 

2/12 

0/12 

0/12 

0/12 

0/11 

T/D Longitudinal 
Position (ft) 

Mean 

923.7 

1016.3 

1097.6 

798.6 

1026.3 

Std Dev 

324.7 

401.9 

431.1 

358.5 

288.9 

Min 

194.1 

396.4 

452.9 

430.0 

706.9 

Max 

1289.8 

1682.6 

1942.5 

1662.7 

1635.4 

Rating 

Desired 

Desired 

Desired 

Desired 

Desired 

T/D Lateral 
Position (ft) 

Mean 

-0.8 

2.4 

-3.7 

-1.2 

0.6 

Std Dev 

8.2 

19.2 

11.4 

14.7 

12.7 

Min 

-10.9 

-40.5 

-31.8 

-30.8 

-16.0 

Max 

14.9 

26.3 

11.5 

25.9 

19.7 

Rating 

Desired 

Desired 

Desired 

Desired 

Desired 

T/D Sink Rate 
(fps) 

Mean 

-8.1 

-6.6 

-6.7 

-7.7 

-7.6 

Std Dev 

2.7 

3.2 

1.9 

3.7 

3.0 

Min 

-4.0 

-0.9 

-3.5 

-3.4 

-4.5 

Max 

-11.9 

-11.9 

-9.6 

-15.8 

-12.7 

Rating 

Adequate 

Adequate 

Adequate 

Adequate 

Adequate 


The data show that, on average, all five EFVS operational concepts: 

• Were within the “Desired” landing position performance criteria (laterally within ±27 ft of 
centerline and longitudinally between 750 to 2250 ft from threshold) as defined in Table 6. 

• Had higher than expected touchdown sink rates with only “Adequate” sink rate performance 
criteria (between six to 10 feet/second) being met as defined in Table 6. 

AN OVA Analyses 

ANOVAs revealed no significant differences (p>0.05) for touchdown longitudinal position 
(mean=973 ft, o= 370 ft), lateral position (mean=-0.55 ft, g= 13.6 ft), and sink rate (mean=-7.3 ft/sec, 
g= 2.9 ft/sec) for the five EFVS operational concepts tested. 

Objective Landing Standards Analysis 

In Figure 1 1, the touchdown data are shown, for the four EFVS display concepts and the EFVS 
operational baseline concept. Included on this plot is the touchdown aim point (0 ft lateral distance, 
1000 ft longitudinal distance) indicated by the axes origin and the landing box (laterally within ±58 ft 
of centerline and longitudinally between 200 to 2700 ft from threshold) indicated by the dashed 
rectangle. In Figure 12, the pitch angle and bank angle at touchdown are shown. Included on this 
plot are the normal landing ground contact angles for the simulated aircraft indicated by the solid- 
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Figure 11. Touchdown Footprint for EFVS Operational concepts 

Visual inspection of the data shows that 1) all EFVS display concepts were within the landing box 
defined in AC-120-28D and JAR AWO (i.e., within ±58 ft lateral distance of centerline and between 
200 ft to 2700 ft longitudinal distance from threshold); 2) all but one of the Conventional HDD 
concept (EFVS operational baseline concept) landings was within the landing box 3) all EFVS 
Operational concepts, including EFVS operational baseline concept, landed on the runway; 4) there 
were no wing tip or tail strikes. 

None of the five EFVS operational concepts tested met the 10' 6 probability of occurrence auto-land 
criteria for T/D longitudinal position, lateral position from centerline, or sink rate as per AC120-28D 
[34, Appendix 3]. 
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Figure 12. Touchdown Pitch angle and Bank Angle Footprint for EFVS Operational concepts 


4.2.4 Rollout Performance 

Rollout statistics (RMS lateral deviation from centerline and maximum lateral deviation from 
centerline) for the five EFVS operational concepts tested are provided in Table 16. 


Table 16. Rollout Statistics for EFVS Operational Concepts 




Conventional 

HDD 

With 

TDZ/CL 

Lights 

EFVS HUD 

EFVS HUD 

EFVS HUD 

EFVS HUD 



Without 

With 

Without 

With 



TDZ/CL 

TDZ/CL 

TDZ/CL 

TDZ/CL 



Lights 

Lights 

Lights 

Lights 



1800ft RVR/ 

1400 ft RVR/ 

1400 ft RVR/ 

1000 ft RVR/ 

1000 ft RVR/ 



200 ft DH 

150 ft DH 

150 ft DH 

150 ft DH 

150 ft DH 

RMS Lateral 
Deviation from 

Mean 

4.4 

7.8 

6.7 

7.3 

9.2 

Std Dev 

1.9 

4.1 

4.2 

4.0 

3.3 

Centerline (feet) 

Maximum Lateral 
Deviation from 

Mean 

10.1 

18.3 

13.5 

14.4 

17.9 

Std Dev 

4.1 

11.7 

8.8 

7.9 

8.5 

Centerline (feet) 


An ANOVA revealed no significant differences (p>0.05) among the five EFVS operational concepts 
tested for RMS lateral deviation from centerline (mean=7.1 ft, <3=3.5 ft) or maximum lateral deviation 
from centerline (mean=14.9 ft, g= 8.2 ft) during rollout operations. 
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4.2.5 Pilot Workload 


An ANOVA revealed significant differences (F(4,52)=3.912, p=0.008) between the EFVS operational 
concepts for the PF post-run workload ratings. SNK post hoc tests revealed that the Conventional 
HDD concept (mean=3.6) was rated as having significantly higher workload than the four EFVS 
display concepts (mean=2.8). There were no appreciable PF workload differences among the four 
EFVS display concepts tested. 

An ANOVA revealed no significant differences (p>0.05) in PM post-run workload ratings 
(mean=2.6) among the five EFVS operational concepts tested. 

Crews rated their workload as being “moderate, easily managed and having considerable spare time” 
while using the EFVS display concept on approach through landing in visibilities as low as 1000 ft 
RVR. 

4. 2. 6 Eye Tracking 

Eye tracking analysis for the EFVS operational concepts are broken into two main quantitative 
metrics: Head-up percent and transition count from instruments to OTW by segment of approach and 
landing. EFVS operational concepts utilized both the HUD and HDD display locations. Therefore, 
analysis of OTW transition height above threshold and gaze direction was not performed. 

Statistically significant effects are circled in red in Figures 13 and 14. 

Comparing the visual behavior across the EFVS operational concept display conditions, Tukey HSD 
post hoc tests revealed there was a predictable significant effect (Figure 13) for head up percentage 
time between the Conventional HDD concept (EFVS operational baseline concept) and all other 
EFVS HUD concepts for all in-flight segments (Instrument, Transition, and Visual). Of note, during 
the visual segment when pilot attention is expected to be OTW to make the visual approach (DH to 
50 ft HAT) pilots operating with the Conventional HDD condition remained inside the flight deck 
looking at instrumentation over 35% of the time (Figure 13). There was no significant difference 
between any of the conditions for the flare or landing segments. During the flare segment (50 ft HAT 
to touchdown) the runway environment and threshold lights were visible, drawing pilot attention 
nearly completely OTW. Pilot head up time varied between 77% and 84% during the landing rollout, 
indicating pilot attention remained inside the flight deck approximately 16-23% of the time, 
referencing airspeed and the moving map display available on the ND. Table 17 shows the results for 
head up percentage of the three EFVS operational concepts by eye tracking analysis segments. 
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Segment Segment 


Display/RVR Condition 


-♦—Conventional HDD 
1SG0 RVR/200 DH 

-■-EFVS HUD 1400 
RVR/150 DH 

-♦—EFVS HUD 1000 
RVR/150 DH 


Figure 13. EFVS Op Concept, Head Up Time (%) 


Table 17. EFVS Op Concepts, Head Up Time (%) 



Conventional 

HDD 

EFVS HUD 

EFVS HUD 

ANOVA 

1800 RVR/ 
200 DH 

1400 RVR/ 
150 DH 

1000 RVR/ 
150 DH 


Instrument 

Segment 

Mean 

24.3 

95.0 

95.6 

Display/RVR Condition 
F(2,51) = 298.13, p< 0.000 

Std Dev 

16.1 

4.1 

3.6 

Transition 

Segment 

Mean 

44.9 

96.3 

96.8 

Display/RVR Condition 
F(2,51) = 73.46, p < 0.000 

Std Dev 

24.8 

5.3 

3.6 

Visual 

Segment 

Mean 

65.5 

94.2 

98.7 

Display/RVR Condition 
F(2,51) = 14.81, p < 0.000 

Std Dev 

29.4 

14.3 

1.9 

Flare 

Segment 

Mean 

92.4 

91.4 

95.5 

No Significance 

Std Dev 

12.8 

15.8 

10.6 

Landing 

Segment 

Mean 

83.4 

77.8 

83.4 

No Significance 

Std Dev 

14.9 

17.3 

13.5 


There was a significant difference (Figure 18) in the number of visual transitions between the 
Conventional HDD and all of the EFVS conditions for all segments of flight. The number of 
Conventional HDD visual transitions during the transition and visual segments were nearly the same 
with an average near four, verifying pilots’ attention continues to transition between instruments and 
OTW during the visual segment (Figure 18) to check airspeed, guidance and attitude indicators even 
though the approach lights had been acquired to continue the approach. When equipped with a HUD, 
pilots make at most one or no transitions inside the flight deck during the transition and visual 
segments, with all information necessary to complete the approach to land available on the HUD. 
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During the flare segment, transition count for pilots operating under the Conventional HDD concept 
reduces significantly to one or less, with nearly all attention focused OTW. 

During the landing and rollout, pilots began to transition back inside the flight deck, confirming the 
behavior observed with head up time. Transitions were observed with the use of both the EFVS HUD 
and Conventional HDD during the landing segment. This behavior is likely due to additional 
information available on the moving map display aiding the pilot awareness of aircraft position on the 
runway relative to the runway exit taxiway. Tukey HSD post hoc tests indicate the significant 
difference across EFVS operational concepts is due to variance in OTW RVR, suggesting reduced 
RVR below 1400 feet limits pilot transitions between head up and head down. However, head up 
percentages were similar across all conditions suggesting these transitions were for short glances to 
information available head down. Table 18 shows the results for head up/head down transitions of 
the three EFVS operational concepts by eye tracking analysis segments. 
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Table 18. EFVS Op Concepts, Head Up/Head Down Transitions 



Conventional 

HDD 

EFVS HUD 

EFVS HUD 

ANOVA 

1800 RVR/ 
200 DH 

1400 RVR / 
150 DH 

1000 RVR/ 
150 DH 


Instrument 

Segment 

Mean 

11.6 

3.8 

4.1 

Display/RVR Condition 
F(2,51) = 73.46, p < 0.000 

Std Dev 

9.4 

4.0 

4.1 

Transition 

Segment 

Mean 

3.3 

0.5 

0.6 

Display/RVR Condition 
F(2,51) = 12.97, p< 0.000 

Std Dev 

2.6 

1.0 

1.1 

Visual 

Segment 

Mean 

4.2 

0.4 

0.3 

Display/RVR Condition 
F(2,51) = 31.41, p< 0.000 

Std Dev 

2.6 

0.8 

0.7 

Flare 

Segment 

Mean 

1.0 

0.1 

0.2 

No Significance 

Std Dev 

1.8 

0.3 

0.5 

Landing 

Segment 

Mean 

5.3 

6.0 

2.9 

No Significance 

Std Dev 

3.8 

4.4 

2.5 


4.3 SVS Operational Concept Comparisons 

4 . 3. 1 Approach Performance - Instrument Segment 

The five concepts used for SVS operational concept comparisons for approach performance during 
the instrument segment (from 1000 ft to DH) were: 

1) Conventional HUD without TDZ/CL lights (1400 ft RVR/150 ft DH) 

2) SVS HUD (1400 ft RVR/150 ft DH) 

3) SVS HUD (1000 ft RVR/150 ft DH) 

4) SVS HDD (1800 ft RVR/150 ft DH) 

5) SVS HDD (1400 ft RVR/150 ft DH) 

The Conventional HUD flown in 1400 ft RVR to a 150 ft DH and a runway without TDZ/CL lights 
was considered the operational baseline concept for the SVS operational concept comparisons. 
Throughout the remainder of this report, it will be referred to as one of the SVS operational concepts 
tested in this experiment. 

The SVS display runs evaluated for the 1000, 1400, and 1800 ft RVR conditions included the runs 
made with and without TDZ/CL lights. In the visibility conditions tested, the TDZ/CL lights were 
not visible to the crew during the instrument segment of the approach. 

Approach Statistics 


In Table 19, the approach statistics (mean, standard deviation, minimum value, and maximum value) 
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are shown for the five SVS operational concepts. Also provided in Table 19 are the number of runs 
that resulted in a go-around and the total number of runs for the SVS operational concepts tested. 
Table 20 provides event description data gathered from video review of the go-around runs for the 
five SVS Operational Concepts tested. 


Table 19. Approach Statistics (1000 ft to DH) for SVS Operational Concepts 



Conventional 
HUD Without 
TDZ/CL Lights 

SVS HUD 

SVS HDD 

1400 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150 ft DH 

1000 ft RVR/ 
150 ft DH 

1800 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150 ft DH 

Number of Go-Around / 
Number of Runs 

1/12 

4/24 

7/24 

1/24 

4/24 

RMS Localizer 
Deviation (dots) 

Mean 

0.05 

0.05 

0.04 

0.05 

0.04 

Std Dev 

0.02 

0.02 

0.02 

0.03 

0.02 

Min 

0.03 

0.02 

0.02 

0.01 

0.01 

Max 

0.08 

0.09 

0.08 

0.12 

0.10 

RMS Glideslope 
Deviation (dots) 

Mean 

0.11 

0.08 

0.09 

0.26 

0.25 

Std Dev 

0.10 

0.04 

0.04 

0.13 

0.16 

Min 

0.04 

0.03 

0.03 

0.08 

0.09 

Max 

0.38 

0.19 

0.20 

0.64 

0.68 

RMS Sink Rate 
Deviation (fpm) 

Mean 

93.2 

61.1 

67.1 

141.8 

134.1 

Std Dev 

104.4 

16.4 

15.6 

66.8 

82.8 

Min 

39.7 

28.4 

31.4 

56.9 

58.7 

Max 

399.4 

85.6 

95.8 

358.9 

393.9 
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Table 20. Go-Around 1 

3vent Description with SVS Operational Concepts 

Display Concept 

TDZ/CL Lights 

OTW RVR (ft) 

Event Description 

Conventional 

HUD 

OFF 

1400 

PM called 'lights' and then PF called 'lights.' PF did not see runway environment 
at minimums so he performed a go-around. PM could see runway enviornment 
and said 'you are right down the centerline' as the PF called for go-around. 

SVS HUD 

OFF 

1400 

PF turned imagery off shortly before 'minimums' call. PF called 'rabbits', but 
then called 'go-around'. Post-run PF commented that he had threshold lights 
but did not feel comfortable continuing landing with just threshold lights. 

SVS HUD 

ON 

1400 

PF turned imagery OFF/ON near 'approaching minimums' and 'minimums' calls. 
PM called 'lights', followed by PF calling 'lights.' PF called 'going around' as he 
didn't have threshold lights. 

SVS HUD 

OFF 

1400 

PF never turned imagery off. PF called 'nothing, going around' as he did not see 
threshold lights. 

SVS HUD 

ON 

1400 

PF never turned imagery off. PM called 'lights', but PF did not. PF called 'going 
around.' 

SVS HUD 

ON 

1000 

PF turned imagery off shortly before 'minimums' call. PF saw nothing and 
performed go-around at minimums. 

SVS HUD 

OFF 

100 

PF turned imagery off shortly before 'approaching minimums' call. PM called 
'rabbits', followed by PF calling 'rabbits', and then PM called 'lights'. PF called 
'go around' when he didn't see lights by 'minimums' call. 

SVS HUD 

OFF 

1000 

PF turned imagery off at 500 ft AGL. PM called 'lights' and then PF called 
'lights'. PM saw threshold lights but PF did not so called go-around. 

SVS HUD 

OFF 

1000 

After descending through 400 ft AGL, PF turned imagery on/off every 20 feet or 
so. PM called 'lights'. With crosswind and not seeing lights, PF decided to go- 
around at 200 ft AGL. 

SVS HUD 

ON 

1000 

PF never turned imagery off as he was trying to look through SV image. PM 
called 'lights'. PF did not see lights so he performed a go-around. 

SVS HUD 

OFF 

1000 

PF never turned imagery off as he was trying to see lights in blank area of SV 
image. PM called 'lights.' PF did not see lights so he performed a go-around. 

SVS HUD 

ON 

1000 

PF never turned imagery off. PM called 'lights', and then PF called 'lights.' PM 
called 'field in sight', but PF did not see threshold lights at minimums so 
performed go-around. 

SVS HDD 

OFF 

1800 

PF called 'lights, landing'. PF went around at ~35 ft AGL. [Video review showed 
aircraft 2 dots high prior to go-around call.] 

SVS HDD 

ON 

1400 

PF called 'rabbits.' PM called 'you are high' and PF called 'going around.' Post 
run PF commented that he didn't see enough to land. 

SVS HDD 

OFF 

1400 

PM called 'lights’ and then PF called 'lights.' PM called 'runway' and PF called 
'landing'. [Video review showed aircraft was 1 dot low and unstable in roll]. PF 
called go-around at 5 ft AGL. 

SVS HDD 

ON 

1400 

PM called 'lights' and then PF called 'lights.' PM called 'runway' and PF called 
'go around' at minimums because he didn't have threshold lights. 

SVS HDD 

ON 

1400 

PM called 'lights' and then PF called 'lights'. PF called 'landing', but head-down 
to head-up transition and pilot-induced oscillations in roll caused instability. PF 
called 'go-around' at 25 ft AGL. 


The data appears to indicate that the SVS imagery brightness setting may have occluded the approach 
lights for some of the pilots, but not all. The pilots were trained on how to best set the 
brightness/contrast of the SVS image. They were allowed to adjust it at any time during the test and 
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the principal investigator specifically had them set it at the beginning of the day to their personal 
preferences. Comparing the SVS HUD go-around numbers in Table 19 to the EFVS go-around 
numbers in Table 8, the EFVS (FLIR imagery) appears to have enhanced the approach lights for the 
pilot enabling him to line up and see the required visual cues for landing; while the SVS imagery may 
have occluded the approach lights. 

RVR appears to have had an influence of the PF’s decision to land or go-around while using the SVS 
HUD. In the 1000 ft RVR condition, seven of the possible 24 SVS HUD runs resulted in a go-around 
while in the 1400 ft RVR condition four of the possible 24 SVS HUD runs resulted in a go-around. 

RVR also appears to have had an influence of the PF’s decision to land or go-around while using the 
SVS HDD. In the 1800 ft RVR condition, one of the possible 24 SVS HDD runs resulted in a go- 
around while in the 1400 ft RVR condition four of the possible 24 SVS HDD runs resulted in a go- 
around. 

AN OVA Analyses 

An ANOVA revealed that there were no significant (p>0.05) differences for RMS localizer deviation 
(mean=0.05 dots with a=0.02 dots) among the five SVS operational concepts. 

An ANOVA revealed significant differences (F(4,86)=13.446, p<0.0001) between the SVS 
operational concepts for RMS glideslope deviation. SNK post hoc tests indicated the RMS glideslope 
deviation was significantly less for the HUD concepts (Conventional and SVS) compared to the HDD 
SVS concepts. For RMS glideslope deviation, the HUD concepts had a mean=0.09 dots with o= 0.05 
dots and the HDD concepts had a mean=0.26 dots with g= 0.15 dots. 

An ANOVA revealed significant differences (F(4,86)=6.9445, p<0.0001) between the SVS 
operational concepts for RMS sink rate deviation (from a nominal 3-degree glideslope value). SNK 
post hoc tests showed the RMS sink rate deviation was significantly less for the SVS HUD concepts 
(mean=64 fpm and a=16 fpm) compared to SVS HDD concepts (mean=138 fpm and a=75 fpm). 
There were no significant differences between the Conventional HUD concept (mean= 93 fpm and 
g= 104 fpm) and the SVS HUD concepts or the Conventional HUD concept and the SVS HDD 
concepts. 

Objective Approach Standards Analysis 

Existing performance-based approach standards were also applied in the objective data analysis for 
those SVS operational concept approaches that were flown to touchdown. 

The percentage of approaches which met the localizer and glideslope criteria of the four existing 
approach performance standards are shown, broken down by SVS operational concept in Tables 21 
and 22. Note that in Tables 21 and 22 the JAR AWO values were based on the percentage of runs 
flown which met the 1/3 dot localizer and 1 dot glideslope criteria between 300 ft and 100 ft AFF and 
did not include the statistical analysis to ensure a 95% bound as per JAR-AWO. To apply this 
additional constraint on the data, the Continuous Method [32] technique was used to calculate the 
probability of success, P(a), of meeting the AWO exceedance criteria (1/3 dot localizer, 1 dot 
glideslope) 95% of the time with required levels of confidence for the SVS operational concepts 
flown. The calculated P(a) values are shown in Table 23, broken down by SVS operational concept. 
The influence of DA is shown in Tables 21-23, where, if the criteria window included “DA” (e.g., in 
PTS and JAR AWO), the analysis used a corresponding altitude window down to 150 ft. 
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Table 21. Percentage SVS operatonal concept approaches successfully meeting localizer performance 

standards 



Localizer 

Conventional HUD 
without TDZ/CL 
lights 

SVS HUD 

SVS HDD 


1400 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150 ftDH 

1000 ft RVR/ 
150 ftDH 

1800 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150 ft DH 

FAA-S-8081-4D 

100% 

100% 

100% 

100% 

100% 

AC 120-29 

100% 

100% 

100% 

100% 

100% 

FAR 91 

100% 

100% 

100% 

100% 

100% 

JAR-AWO 

100% 

100% 

100% 

100% 

100% 


Table 22. Percentage SVS operatonal concept approaches successfully meeting glideslope performance 

standards 



Glideslope 

Conventional 
HUD without 
TDZ/CL lights 

SVS HUD 

SVS HDD 


1400 ft RVR/ 
150 ft DH 

1400 ft RVR/ 
150 ftDH 

1000 ft RVR/ 
150 ft DH 

1800 ft RVR/ 
150 ftDH 

1400 ft RVR/ 
150 ft DH 

FAA-S-8081-4D 

100% 

100% 

100% 

87% 

90% 

AC 120-29 

91% 

100% 

100% 

91% 

100% 

FAR 91 

73% 

90% 

94% 

48% 

55% 

JAR-AWO 

82% 

90% 

94% 

48% 

55% 


Table 23. Probability of Success in meeting AWO exceedance critieria with SVS operational concepts 


Display Concept 

RVR/DH 

(ft) 

Localizer 

P(a) 

Glideslope 

P(a) 

Number 
of Runs 

Number of 
Go-Around 

Conventional HUD without TDZ/CL Lights 

1400/150 

100 

66 

12 

1 

SVS HDD 

1400/150 

100 

39 

24 

4 

SVS HDD 

1800/150 

99 

30 

24 

1 

SVS HUD 

1000/150 

100 

96 

24 

7 

SVS HUD 

1400/150 

100 

88 

24 

4 


The data indicate that: 

• Localizer tracking was never an issue, irrespective of the criteria. 

• Glideslope tracking performance appears to be affected by display location (head-up versus 
head-down) and the presence of SVS imagery. 

• SVS HUD in 1000 ft RVR condition was able to meet the JAR- AWO localizer and glideslope 
criteria but with a 29% (7 out of 24 runs) missed approach rate. 
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4.3.2 Approach Performance - Visual Segment 

The nine concepts used for SVS operational concept comparisons for approach performance during 
the visual segment (from DH to threshold crossing) were: 

1) Conventional HUD without TDZ/CL lights (1400 ft RVR/150 ft DH) 

2) SVS HUD with TDZ/CL lights (1400 ft RVR/150 ft DH) 

3) SVS HUD without TDZ/CL lights (1400 ft RVR/150 ft DH) 

4) SVS HUD with TDZ/CL lights (1000 ft RVR/150 ft DH) 

5) SVS HUD without TDZ/CL lights (1000 ft RVR/150 ft DH) 

6) SVS HDD with TDZ/CL lights (1800 ft RVR/150 ft DH) 

7) SVS HDD without TDZ/CL lights (1800 ft RVR/150 ft DH) 

8) SVS HDD with TDZ/CL lights and (1400 ft RVR/150 ft DH) 

9) SVS HDD without TDZ/CL lights (1400 ft RVR/150 ft DH) 

The Conventional HUD flown in 1400 ft RVR to a 150 ft DH and a runway without TDZ/CL lights 
was considered the operational baseline concept for the SVS operational concept comparisons. 

In Tables 24-26, statistics of lateral path performance, vertical path performance, and sink rate 
deviation at 100 ft HAT and at threshold crossing are provided to evaluate how effectively the pilots 
could use the different SVS operational concepts during the visual portion of the approach segment. 
The maximum bank angle from threshold crossing to touchdown and the bank angle at touchdown 
were also used to evaluate the effectiveness of the different SVS operational concepts, including the 
operational baseline, and are presented in Table 27. 
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Table 24. Lateral Path Performance Statistics of SVS Operational Concepts in the Visual Segment of the 

Approach 



Conventional 
HUD 
Without 
TDZ/CL Lights 

SVS HUD 
With 
TDZ/CL 
Lights 


SVS HUD 
With 
TDZ/CL 
Lights 


SVS HDD 
With 
TDZ/CL 
Lights 

SVS HDD 
Without 
TDZ/CL 
Lights 

SVS HDD 
With 
TDZ/CL 
Lights 

SVS HDD 
Without 
TDZ/CL 
Lights 

1400ft RVR 

1400 ft 
RVR 

1400ft 

RVR 

1000 ft 
RVR 

1000ft 

RVR 

1800 ft 
RVR 

1800 ft 
RVR 

1400 ft 
RVR 

1400 ft 
RVR 

No. Go-Around / 
No. Runs 

1/12 

2/12 

2/12 

3/12 

4/12 

1/12 

0/12 

3/12 

1/12 

Lat Path Dev at 
100 ft HAT (ft) 

Mean 

- 2.6 

- 1.2 

- 13.1 

3.5 

- 5.3 

- 7.7 


- 1.7 

- 3.5 

Std 

Dev 

11.5 

18.0 

18.9 

8.7 

13.3 

23.7 

20.2 

13.0 

19.4 

Max 

Left 

- 16.8 

- 22.0 

- 54.6 

- 12.5 

- 27.5 

- 58.2 

- 35.2 

- 24.4 

- 34.9 

Max 

Right 

23.2 

24.2 

14.4 

11.2 

14.4 

28.5 

33.3 

15.4 

32.4 

Lat Path Dev at Threshold 
Crossing (ft) 

Mean 

- 3.0 

0.5 

- 10.4 

4.4 

L 

oo 

- 0.4 

- 6.8 

- 9.9 

0.7 

Std 

Dev 

16.3 

16.0 

20.0 

12.3 

15.0 

16.1 

16.5 

14.5 

20.4 

Max 

Left 

- 26.5 

- 24.4 

- 47.2 

- 12.1 

- 29.1 

- 36.1 

- 45.1 

- 39.9 

- 25.2 

Max 

Right 

25.8 

25.0 

23.5 

31.0 

13.9 

21.1 

12.7 

9.7 

52.7 
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Table 25. Vertical Path Performance Statistics of SVS Operational Concepts in the Visual Segment of the 

Approach 



Conventional 
HUD 
Without 
TDZ/CL Lights 

SVS HUD 

With 

TDZ/CL 

Lights 

SVS HUD 
Without 
TDZ/CL 
Lights 

SVS HUD 

With 

TDZ/CL 

Lights 

SVS HUD 
Without 
TDZ/CL 
Lights 

SVS HDD 

With 

TDZ/CL 

Lights 


SVS HDD 
With 
TDZ/CL 
Lights 

SVS HDD 
Without 
TDZ/CL 
Lights 

1400 ft RVR 

1400 ft 
RVR 

1400 ft 
RVR 

1000 ft 
RVR 

1000 ft 
RVR 

1800 ft 
RVR 

EH 

1400 ft 
RVR 

1400 ft 
RVR 

No. Go-Around/ 
No. Runs 

1/12 

2/12 

2/12 

3/12 

4/12 

1/12 

0/12 

3/12 

1/12 

+■» 

Mean 

-3.8 

-1.4 

-2.7 

-3.6 

-8.7 


6.7 

-3.3 

1.3 

Vert Path Dev 
100 ft HAT (ft 

Std Dev 

7.1 

5.1 

9.0 

4.0 

5.6 

16.5 

14.7 

13.7 

22.0 

Max 

Below 

-11.6 

-7.2 

-9.6 

-10.3 

-13.6 

-30.6 

-13.8 

-29.0 

-22.0 

Max 

Above 

13.7 

9.2 

22.4 

3.4 

3.5 

23.5 

26.8 

15.7 

39.1 

Altitude at Threshold 
Crossing (ft) 

Mean 

51.5 

52.6 

50.8 

45.4 

51.3 

52.9 

52.9 

52.2 

54.9 

Std Dev 

10.2 

5.0 

8.8 

4.0 

12.8 

16.7 

16.7 

17.0 

21.9 

Min 

40.8 

46.3 

43.3 

35.9 

37.5 

19.0 

19.0 

25.3 

26.4 

Max 

69.9 

61.6 

72.3 

49.1 

75.4 

74.0 

74.0 

76.1 

90.2 
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Table 26. Sink Rate Statistics of SYS Operational Concepts in the Visual Segment of the Approach 



Conventional 
HUD 
Without 
TDZ/CL Lights 

SVS HUD 

With 

TDZ/CL 

Lights 

SVS HUD 
Without 
TDZ/CL 
Lights 

SVS HUD 

With 

TDZ/CL 

Lights 

SVS HUD 
Without 
TDZ/CL 
Lights 

SVS HDD 

With 

TDZ/CL 

Lights 

SVS HDD 
Without 
TDZ/CL 
Lights 

SVS HDD 
With 
TDZ/CL 
Lights 

SVS HDD 
Without 
TDZ/CL 
Lights 

1400 ft RVR 

1400 ft 
RVR 

1400 ft 
RVR 

1000 ft 
RVR 

1000 ft 
RVR 

1800 ft 
RVR 

1800 ft 
RVR 

1400 ft 
RVR 

1400 ft 
RVR 

No. Go-Around/ 
No. Runs 

1/12 

2/12 

2/12 

3/12 

4/12 

1/12 

0/12 

3/12 

1/12 

Sink Rate at 100 ft HAT 
(ft/min) 

Mean 

-683.6 

-753.9 

-711.6 

-782.2 

-675.1 

-743.4 

-647.3 

-721.5 

-749.4 

Std 

Dev 

85.4 

93.1 

79.0 

28.5 

113.7 

218.5 

118.7 

232.3 

182.2 

Max 

-815.6 

-952.7 

-806.4 

-829.0 

-869.5 

-1168.2 

-871.6 

-1159.3 

-1046.0 

Sink Rate at Threshold 
Crossing (ft/min) 

Mean 

-576.7 

-562.0 

-632.9 

-676.3 

-545.1 

-600.0 

-673.2 

-535.9 

-555.1 

Std 

Dev 

124.1 

188.1 

121.5 

112.7 

249.9 

253.0 

165.6 

190.7 

297.7 

Max 

-735.8 

-764.9 

-829.6 

-871.6 

-788.3 

-896.1 

-915.2 

-793.3 

-1014.6 


ANOVA analyses indicate no significant differences (p>0.05) for lateral deviation from centerline, 
vertical deviation from path, or sink rate among the nine SVS operational concepts tested at both the 
100 ft HAT and Threshold Crossing Locations. 
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Table 27. Bank Angle Statistics of SYS Operational Concepts 



Conventi 
onal HUD 
Without 
TDZ/CL 
Lights 

SVS HUD 

With 

TDZ/CL 

Lights 

SVS HUD 
Without 
TDZ/CL 
Lights 

SVS HUD 
With 
TDZ/CL 
Lights 

SVS HUD 

Without 

TDZ/CL 

Lights 

SVS HDD 
With 
TDZ/CL 
Lights 

SVS HDD 
Without 
TDZ/CL 
Lights 

SVS HDD 
With 
TDZ/CL 
Lights 

SVS HDD 
Without 
TDZ/CL 
Lights 

1400 ft 
RVR 

1400 ft 
RVR 

1400 ft 
RVR 

1000 ft 
RVR 

1000ft 

RVR 

1800 ft 
RVR 

1800 ft 
RVR 

1400 ft 
RVR 

1400 ft 
RVR 

£ 2 _ 

2 W) W) 
M- c <u 
CD "O 

Mean 

3.9 

3.5 

3.7 

3.0 

3.1 

n 

■ 

5.2 

3.8 

f 51 

J 

a 5 | 

i/) 3 

X 0 o 
( 0 * 1 - 

2 iE 

Std 

Dev 

2.8 

3.1 

D 

1.9 

3.5 

2.6 

3.0 

2.9 

2.6 

Max 

7.1 

10.0 

9.8 

6.3 

11.2 

9.0 

10.4 

11.0 

7.3 

Bank Angle at 
Touchdown (deg) 

Mean 

D 

2.6 

2.6 

1.4 

2.7 

2.6 

■ 

3.2 

2.0 

Std 

Dev 

2.8 

3.2 

2.7 

0.9 

3.7 

2.6 

2.2 

■ 

2.7 

Max 

7.2 

10.1 

7.8 

2.8 

11.1 

9.0 

8.2 

7.7 

7.5 


ANOVA analyses indicate no significant differences (p>0.05) among the nine SVS operational 
concepts for the maximum bank angle from threshold to touchdown or for the bank angle at 
touchdown measures. 

4.3.3 Landing Performance 

The nine concepts used for SVS operational concept comparisons for landing performance were: 

1) Conventional HUD without TDZ/CL lights (1400 ft RVR/150 ft DH) 

2) SVS HUD with TDZ/CL lights (1400 ft RVR/150 ft DH) 

3) SVS HUD without TDZ/CL lights (1400 ft RVR/150 ft DH) 

4) SVS HUD with TDZ/CL lights (1000 ft RVR/150 ft DH) 

5) SVS HUD without TDZ/CL lights (1000 ft RVR/150 ft DH) 

6) SVS HDD with TDZ/CL lights (1800 ft RVR/150 ft DH) 

7) SVS HDD without TDZ/CL lights (1800 ft RVR/150 ft DH) 

8) SVS HDD with TDZ/CL lights and (1400 ft RVR/150 ft DH) 

9) SVS HDD without TDZ/CL lights (1400 ft RVR/150 ft DH) 
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The Conventional HUD flown in 1400 ft RVR to a 150 ft DH and a runway without TDZ/CL lights 
was considered the operational baseline concept for the SVS operational concept comparisons. 

Landing Statistics 

In Table 28, the T/D statistics (mean, standard deviation, minimum value, and maximum value) are 
shown, broken out by RVR and TDZ/CL light configuration, for the nine SVS operational concepts. 
The T/D measures (lateral position, longitudinal position and sink rate) means were used to determine 
which touchdown performance rating level (Desired, Adequate, or Not Adequate) as defined in Table 
6 was achieved for the SVS operational concepts. Also provided in Table 28 are the number of runs 
that resulted in a go-around and the total number of runs for the SVS operational concepts tested. 

Note that for touchdown lateral position statistics in Table 28, the “min” value equates to the 
maximum deviation to the left of centerline and the “max” value equates to the maximum deviation to 
the right of centerline. 


Table 28. Landing Statistics for SVS Operational Concepts 



Conventional 
HUD Without 
TDZ/CL Lights 

SVS 

HUD 

With 

TDZ/CL 

Lights 

SVS 

HUD 

Without 

TDZ/CL 

Lights 

SVS 

HUD 

With 

TDZ/CL 

Lights 

SVS 

HUD 

Without 

TDZ/CL 

Lights 

SVS 

HDD 

With 

TDZ/CL 

Lights 

SVS 

HDD 

Without 

TDZ/CL 

Lights 

SVS 

HDD 

With 

TDZ/CL 

Lights 

SVS 

HDD 

Without 

TDZ/CL 

Lights 

1400ft 

RVR 

1400ft 

RVR 

1400ft 

RVR 

1000ft 

RVR 

1000ft 

RVR 

1800ft 

RVR 

1800ft 

RVR 

1400ft 

RVR 

1400ft 

RVR 

No.Go-Around 
/ No. Runs 

1/12 

2/12 

2/12 

3/12 

4/12 

1/12 

0/12 

3/12 

1/12 

Mean 

639.1 

1183.8 

840.9 

770.8 

742.0 

944.0 

835.1 

1003.3 

328.3 
452.5 
1477.8 
Desired 

685.0 

.E ~ Std 

"i ^ Dev 

275.9 

450.1 

283.8 

157.7 

496.6 

546.1 

397.2 

389.8 

‘Eibo ... 

c ' 4 = Min 

o 's; .... 

369.7 

605.9 

457.7 

462.5 

229.9 

- 50.0 

341.1 

26.8 

— 1 o 

Q a - Max 

1243.6 

2022.1 

1330.3 

948.9 

1814.2 

2146.5 

1711.7 

1244.0 

H 

Rating 

Adequate 

Desired 

Desired 

Desired 

Adequate 

Desired 

Desired 

Adequate 

c Mean 

o 

0.5 

2.0 

- 3.3 

5.4 

- 3.3 

5.2 

- 3.5 

- 12.5 

21.6 

2.5 

5 Std 

S. Dev 

13.8 

21.0 

16.6 

11.6 

18.2 

11.0 

19.1 

21.8 

2 £ Min 

<D 

- 14.1 

- 27.9 

- 27.1 

- 15.2 

- 32.4 

- 16.6 

- 33.4 

- 52.4 

18.9 

Desired 

- 33.5 

to 

- 1 Max 

Q 

29.5 

43.7 

19.1 

20.3 

19.2 

46.9 

26.8 

50.7 

H Rating 

Desired 

Desired 

Desired 

Desired 

Desired 

Desired 

Desired 

Desired 

Mean 

- 8.2 

-7A 

- 7.2 

- 6.4 

- 6.8 

- 6.8 

- 7.0 

- 6.6 

3.7 

- 0.7 

- 11.6 

Adequate 

- 6.9 

2 Std 

ns ^ 

cc Dev 

1.8 

2.1 

4.2 

2.0 

2.4 

3.1 

3.2 

2.8 

i /> 

| £ Min 

- 5.7 

- 3.9 

- 0.6 

- 4.3 

- 3.4 

- 2.6 

- 1.8 

- 2.4 

^ Max 

- 10.6 

- 10.7 

- 13.5 

- 9.9 

- 10.3 

- 11.4 

- 12.5 

- 11.7 

Rating 

Adequate 

Adequate 

Adequate 

Adequate 

Adequate 

Adequate 

Adequate 

Adequate 


The data show that: 

• The absence or presence of TDZ/CL lights appears to not have influenced the PF’s decision 
to land or go-around while using the SVS HUD. 


57 


• All HUD (Conventional and SVS) operational concept landings were within the touchdown 
box defined in AC-120-28D [34] and JAR AWO [32] (i.e., within ±58 ft lateral distance of 
centerline and between 200 ft to 2700 ft longitudinal distance from threshold). 

• All SVS operational concept comparison runs met the lateral touchdown criteria (within 57 ft 
of centerline) 

• Unbeknownst to them, one crew landed 50 feet SHORT of the runway while flying with the 
SVS HDD display concept with TDZ/CL lights in the 1800 ft RVR condition. 

• One SVS HDD without TDZ/CL lights in the 1400 ft RVR condition run (one out of 1 1 
landings) did not meet the longitudinal touchdown criteria (between 200 to 2700 ft from 
threshold). The other 10 landings for this operational concept met the criteria. 

• On average, all SVS operational concept comparison runs had higher than expected 
touchdown sink rates with only “Adequate” sink rate performance criteria (between six to 10 
feet/second) being met as defined in Table 6. 

AN OVA Analyses 

ANOVAs revealed no significant differences (p>0.05) for touchdown longitudinal position 
(mean=847 ft, g= 404 ft), lateral position (mean=-0.61 ft, g= 18.4 ft), or sink rate (mean=-7.1 ft/sec, 
g= 2.8 ft/sec) for the nine SVS operational concepts tested. 

Objective Landing Standards Analysis 

In Figure 15, the touchdown data are shown for the eight SVS display concepts and the SVS 
operational baseline concept. Included on this plot is the touchdown aim point (0 ft lateral distance, 
1000 ft longitudinal distance) indicated by the axes origin and the landing box (laterally within ±58 ft 
of centerline and longitudinally between 200 to 2700 ft from threshold) indicated by the dashed 
rectangle. In Figure 16, the pitch angle and bank angle at touchdown are shown. Included on this 
plot are the normal landing ground contact angles for the simulated aircraft indicated by the solid- 
lined contour on the plot. 
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Figure 15. Touchdown Footprint for SYS Operational concepts 
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SVS Baseline 

TDZ C/L Lights, HDD, 1800 RVR 
No TDZ C/L Lights, HDD, 1 800 RVR 
TDZ C/L Lights, HDD, 1400 RVR 
No TDZ C/L Lights, HDD, 1400 RVR 
TDZ C/L Lights, HUD, 1400 RVR 
No TDZ C/L Lights, HUD, 1400 RVR 
TDZ C/L Lights, HUD, 1000 RVR 
No TDZ C/L Lights, HUD, 1000 RVR 



Maximum Ground Contact Angles 


5 10 15 

Absolute Value of Bank Angle, deg 


Figure 16. Touchdown pitch angle and bank angle footprint for SVS operational concepts 
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Visual inspection of the data shows that 1) all HUD (SVS and Conventional) display concepts were 
within the landing box defined in AC-120-28D and JAR AWO (i.e., within ±58 ft lateral distance of 
centerline and between 200 ft to 2700 ft longitudinal distance from threshold); 2) three SVS HDD 
concept landings were outside the landing box, with one (of the three) landing short of the runway 3) 
there were no wing tip or tail strikes, and 4) all SVS Operational concepts, including baseline, had 
higher than expected touchdown sink rates. 

None of the SVS operational concepts tested met the 10" 6 probability of occurrence auto-land criteria 
for T/D longitudinal position, lateral position from centerline, or sink rate as per AC120-28D [34, 
Appendix 3]. 

4 . 3. 4 Rollout Performance 

Rollout statistics (RMS lateral deviation from centerline and maximum lateral deviation from 
centerline) for the nine SVS operational concepts tested are provided in Table 29. 

Separate ANOVAs revealed no significant differences (p>0.05) in PF RMS lateral deviation from 
centerline (mean= 8.6 ft, a=4.4) or maximum lateral deviation from centerline (mean= 19.8 ft, 
a=T0.0) during rollout operations among the nine SVS operational concepts tested. 


Table 29. Rollout Statistics for SVS Operational Concepts 



Conventional 
HUD 
Without 
TDZ/CL Lights 

SVS HUD 
With 
TDZ/CL 
Lights 

SVS 

HUD 

Without 

TDZ/CL 

Lights 

SVS 

HUD 

With 

TDZ/CL 

Lights 

SVS 

HUD 

Without 

TDZ/CL 

Lights 

SVS 

HDD 

With 

TDZ/CL 

Lights 

SVS 

HDD 

Without 

TDZ/CL 

Lights 

SVS 

HDD 

With 

TDZ/CL 

Lights 

SVS 

HDD 

Without 

TDZ/CL 

Lights 

1400ft 

RVR 

1400ft 

RVR 

1400ft 

RVR 

1000ft 

RVR 

1000ft 

RVR 

1800ft 

RVR 

1800ft 

RVR 

1400ft 

RVR 

1400ft 

RVR 

2 £ 

£ £ 

^ 2 
cc 4= 

Mean 

8.2 

8.4 

8.8 

8.2 

9.4 

8.7 

8.6 

8.6 

8.8 

Std 

Dev 

4.1 

5.5 

4.3 

3.8 

4.1 

4.7 

3.4 

6.1 

4.0 

+■» * — * 
.2 £ 

e< 

ro u 
* £ 
i e 

Mean 

18.1 

19.6 

20.2 

15.8 

18.5 

20.2 

21.5 

22.6 

20.9 

Std 

Dev 

9.3 

10.5 

7.6 

6.9 

8.5 

11.0 

8.0 

15.4 

12.5 


4. 3. 5 Pilot Workload 

Separate ANOVAs revealed no significant differences (p>0.05) in PF post-run workload ratings 
(mean=3.3) or PM post-run workload ratings (mean=2.7) among the nine SVS operational concepts 
tested. 

Crews rated their workload as being “moderate, easily managed and having considerable spare time” 
while using the SVS HUD concepts on approach through landing in visibilities as low as 1000 ft 
RVR and SVS HDD concepts in visibilities as low as 1400 ft RVR. 


60 




4. 3. 6 Eye Tracking 

Eye tracking analysis for the SVS operational concepts are broken into two main quantitative metrics; 
Head up percent and transition count from instruments to OTW by segment of approach and landing. 
SVS operational concepts utilized both HUD and HDD display locations; therefore, analysis of OTW 
transition height above threshold and gaze direction was not performed. Statistically significant 
findings are circled in red in Figures 17 and 18. 

ANOVAs performed to assess visual behavior differences across the SVS operational concepts 
indicated significant variance in head up time between HUD and HDD display concepts during all in- 
flight segments of the approach (Figure 17). There were no significant differences in head up 
percentage across conditions during the flare and landing segments. 

Similar behavior to EFVS operational concepts was observed during the in-flight segments of the 
approach, revealing approximately 23% head down time during the visual segment with SVS HDD 
conditions (Table 30). However, these results indicate an increase of 10% head up time over the 
Conventional HDD condition (compare Figure 13 and Figure 17) during the visual segment. 

Tukey HSD pair-wise comparison tests of head up percentage revealed no significant effects due to 
presence of SVS on the HUD contrasted to the Conventional HUD condition across all segments. No 
significant effects were observed due to RVR or TDZ/CL lighting conditions. 



Display/RVR Condition 


-Conventional HUD 


-*-$VS HUD 1400 RVR/ 
ISO DH 

-*-SVSHUD 1000 RVR/ 
150 DH 

— SVS HDD 1800 RVR/ 
150 DH 

-4— SVS HDD 1400 RVR/ 
150 DH 


Segment Segment 


Figure 17. SVS Op Concepts, Head Up Time (%) 
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Table 30. SYS Op Concepts, Head Up Time (%) 



Conventional 

HUD 

SVS HUD 

SVS HUD 

SVS HDD 

SVS HDD 

ANOVA 

1400 RVR/ 
150 DH 

1400 RVR/ 
150 DH 

1000 RVR/ 
150 DH 

1800 RVR/ 
150 DH 

1400 RVR/ 
150 DH 


Instrument 

Segment 

Mean 

95.2 

94.1 

94.1 

18.6 

24.1 

Display/RVR Condition 
F(4,92) = 184.75, 

p < 0.000 

Std 

Dev 

4.3 

8.3 

7.1 

15.7 

20.4 

Transition 

Segment 

Mean 

98.9 

93.0 

94.9 

43.2 

42.9 

Display/RVR Condition 
F(4,91) = 60.70, 

p < 0.000 

Std 

Dev 

1.3 

11.2 

7.1 

19.2 

25.3 

Visual 

Segment 

Mean 

99.5 

93.0 

94.3 

77.5 

75.9 

Display/RVR Condition 
F(4,89) = 6.01, 

p < 0.000 

Std 

Dev 

0.9 

14.4 

13.9 

24.8 

24.2 

Flare 

Segment 

Mean 

99.2 

91.8 

98.8 

94.8 

98.2 

No Significance 

Std 

Dev 

1.4 

17.2 

2.4 

15.2 

4.0 

Landing 

Segment 

Mean 

74.3 

85.3 

85.8 

89.7 

88.9 

No Significance 

Std 

Dev 

16.8 

8.8 

13.8 

9.5 

4.0 


Analysis of head up/head down transitions indicated a significant difference across conditions during 
all in flight segments of the approach to land with the exception of the flare segment. Tukey HSD 
post hoc tests revealed SVS HDD conditions had significantly higher numbers of transitions than 
SVS and Conventional HUD conditions (Figure 31). Analysis showed no significant effect across 
display conditions during the flare segment, averaging nearly zero transitions, indicating pilot 
attention is strictly OTW, further confirmed with results of the head up time analysis (Figure 17). 

ANOVA results of the landing segment transitions showed significant differences across display 
conditions. Tukey HSD post hoc tests indicate these differences are due not to the display location of 
HUD vs. HDD, but the OTW RVR. RVR levels at 1000 ft significantly reduced the number of 
transitions between head up and head down. Reasoning behind pilot visual transitions during the 
landing rollout segment is the information available inside the flight deck, relevant to ownship and 
runway exit ramp information being available on the moving map display. This additional 
information not available on the HUD encourages a division of pilot attention between flight deck 
HDDs and OTW. Data suggests this divided attention is increasingly limited at the lower 1000 RVR 
level with a reduction in visual transitions to the moving map display. This is reasonably due to 
decreased range in OTW visibility that reduces the time to visually acquire any dynamic changes in 
the OTW scene. With the reduction in visual transitions at 1000 RVR, pilots maintained a relatively 
equal (within 10%) amount of time head up as with all other conditions, with the least amount of time 
head up shown with the conventional HUD condition (Figure 17). Although not statistically 
significant, these results suggest the additional information available through the use of SVS aided 
pilots in maintaining attention OTW during the landing segment, even when divided attention was 
limited at lower RVR levels (Figure 18). Table 31 shows specific average transition values across all 
tested conditions. 
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Figure 18. SYS Op Concepts, Flead Up/Down Transitions 
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Table 31. SYS Op Concepts, Head Up/Down Transitions 



Conventional 

HUD 

SVS HUD 

SVS HUD 

SVS HDD 

SVS HDD 

ANOVA 

1400 RVR/ 
150 DH 

1400 RVR/ 
150 DH 

1000 RVR/ 
150 DH 

1800 RVR/ 
150 DH 

1400 RVR/ 
150 DH 


Instrument 

Segment 

Mean 

3.2 

3.2 

4.3 

9.1 

10.3 

Display/RVR 
Condition 
F(4,92) = 5.78, 

p < 0.000 

Std Dev 


3.6 

6.3 

6.2 

9.6 

Transition 

Segment 

Mean 

0.3 

0.5 

0.5 

5.1 

3.7 

Display/RVR 
Condition 
F(4,91) = 31.40, 

p < 0.000 

Std Dev 

0.7 

0.8 

0.9 

2.7 

2.4 

Visual 

Segment 

Mean 

0.1 

0.1 

0.1 

2.1 

2.0 

Display/RVR 
Condition 
F(4,91) = 18.78, 

p < 0.000 

Std Dev 

0.3 

0.3 

0.4 

1.1 

1.8 

Flare 

Segment 

Mean 

0.1 

0.6 

0.2 

0.2 

0.4 

No Significance. 

Std Dev 

0.3 

0.5 

0.5 

0.5 

0.7 

Landing 

Segment 

Mean 

5.8 

3.6 

2.0 

5.4 

4.4 

Display/RVR 
Condition 
F(4,91) = 4.08, 
p < 0.004 

Std Dev 

3.6 

3.3 

2.6 

3.6 

3.6 


4.4 Crew Display Concept Preferences 

The PF and PM were independently asked to rank order the display concepts (Conventional HDD, 
SVS HDD, Conventional HUD, SVS HUD, and EFVS) from most preferred display (rank=l) to least 
preferred display (rank=5) for flying with in low-visibility conditions. Both the PF and PM pilots 
ranked their low- visibility operations display concept preferences as follows: 1) EFVS (most 
preferred), 2) SV HUD, 3) Conventional HUD, 4) SVS HDD, and 5) Conventional HDD (least 
preferred). 

Friedman’s test statistic showed there were significant differences among the displays for both the PF 
(X 2 (4)=36.145, pO.OOl) and PM (x 2 (4)=41.818, p<0.001) preference rankings. Wilcoxon tests were 
used to follow up this finding. A Bonferroni correction was applied and all effects are reported at a 
0.01 level of significance. 

In Figure 19, the inverse mean rankings for the PF and PM display concept preferences are shown. 

By using the inverse of the preference rankings, the most preferred display would have a value of 1 
and the least preferred display would have a value of 0.2 (i.e., 1/5=0. 2) in Figure 19. 
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The PF preferences were significantly better for: 1) EFVS over all other concepts, 2) SVS HUD than 
SVS HDD, and 3) SVS HDD than Conventional HDD. There were no significant PF preference 
differences between: 1) SVS HUD and Conventional HUD or 2) Conventional HUD and SVS HDD. 

The PM preferences were significantly better for: 1) EFVS over all other concepts, 2) HUD concepts 
than HDD concepts, 3) SVS HDD than Conventional HDD. There were no significant PM preference 
differences between SVS HUD and Conventional HUD. 


PF Inverse Mean 
Display Preference Rankings 


SVS HUD 
Conventional HUD 


EFVS 


PM Inverse Mean 
Display Preference Rankings 



SVS HDD 


Conventional HDD 




SVS HUD 


Conventional HUD 


SVS HDD 
Conventional HDD 


0 0.2 0.4 

Least Preferred 
Concept 


0.6 



EFVS 


0.8 


Most Preferred 
Concept 


Figure 19. PF and PM Display Concept Preference Rankings 

4.5 CDTI and SEVS Influences on Traffic Awareness 


Traffic awareness was measured using several methods, both objective and subjective, to evaluate the 
influence of SEVS, CDTI, and their interaction. 

4 . 5 . 1 Traffic Awareness Probe 


A traffic awareness probe, modeled after a SAGAT test, was administered (see Section 3.12 for 
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experimental method and protocol used for this measure). The data quantifies the flight crew’s 
awareness (PF and PM) of traffic and the influence of CDTI. 

The data from the traffic probe are shown in Table 32 and Table 33. In Table 32, the data from the 
first time that the pilots responded to the probe are given. In Table 33, the data from all other queries 
are given. The first and remaining probes are separated because the flight crews were not aware that 
this probe was going to be conducted. Therefore, the first occurrence indicates unexpected behavior 
whereas the other data reflects modified behavior, albeit the probes were not very frequent. 

The data are presented for crew position (PF or PM) and whether CDTI was present or absent. The 
CDTI condition is collapsed across the two different CDTI types (i.e., the moving map with ground 
traffic and moving map with runway inset of ground traffic are treated as not being significantly 
different). 


Table 32. Statistics from First Occurrence of Traffic Probe 




Number 

of 

Responses 


Total 

Number 

of 

Traffic 

% 

Responses 

Min. 

Responses 
per Run 

Max. 

Responses 
per Run 

% Correct 
(Of 

Responses) 

PF 

No CDTI 


3 

out of 

60 


5 

0 

2 

100 


CDTI 


11 

out of 

84 


13 

0 

4 

91 

PM 

No CDTI 


7 

out of 

60 


12 

0 

4 

100 


CDTI 


12 

out of 

84 


14 

0 

3 

100 


Table 33. Statistics from All Other Occurrences of Traffic Probe 




Number 

of 

Responses 


Total 

Number 

of 

Traffic 

% 

Responses 

Min. 

Responses 
per Run 

Max. 

Responses 
per Run 

% Correct 
(Of 

Responses) 

PF 

No CDTI 

34 

out of 

204 


17 


0 

5 


100 


CDTI 

74 

out of 

432 


17 


0 

5 


93 

PM 

No CDTI 

30 

out of 

204 


15 


0 

4 


93 


CDTI 

147 

out of 

432 


34 


0 

11 


95 


These data were not tested for statistically significant differences due to the lack of sufficient 
statistical power. The statistics in Tables 32 and 33 do show several trends, as noted in the following: 

• The percent of responses that were correct when they answered was very nearly 100% for all 
pilots, on all occasions. This indicates that the pilots were not guessing in response to the 
probe. 

• The “first occurrence” data showed that, nominally, pilots were not aware of traffic around 
the active runway. The data show that only 5 to 14% of the traffic was noticed by the pilots 
in the initial administration of the traffic probe. In many cases, the pilots had no awareness of 
the traffic (i.e., as indicated by the “min. responses per run” data, on at least one run within 
each category, the PF or PM could not recall the presence or location of any traffic around the 
active runway). 
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• The data showed that when pilots had expectations that a traffic probe was being 
administered (Table 33) CDTI can aid the PM for traffic awareness. The PM responses with 
CDTI show that the PM was twice as aware of the traffic when CDTI was available 
compared to when it was not. On the other hand, the presence of CDTI did not appear to 
increase the PF’s awareness of traffic. Because this result occurs with “expectation,” it 
implies that pilots must be trained in the use of CDTI. This training should emphasize to the 
pilots to include the CDTI information found on their displays in their visual scan for traffic 
there and OTW. 

• Unfortunately, even with expectation and CDTI, the PMs were only aware of one-third of the 
traffic on average around the active runway and on at least one run, the PM could not recall 
the presence or location of any traffic around the active runway. 

4.5.2 Paired Comparisons for Situation Awareness and Traffic Awareness 

Post-test SA was assessed using the SA-SWORD paired-comparison technique which provided 
relative SA ratings across the three CDTI formats tested. For these comparisons, SA was defined as 
“ The pilot's awareness and understanding of all factors that will contribute to the safe flying of their 
aircraft under normal and non-normal conditions .” The PF and PM independently completed the 
SA-SWORD technique for the CDTI formats tested. 

Similarly, a post-test paired-comparison technique was used to assess traffic awareness for the CDTI 
formats tested. For these comparisons, traffic awareness was defined as “The pilot's awareness and 
understanding of significant traffic that will affect his/her aircraft under normal and non-normal 
operating conditions .” The PF and PM independently completed the traffic awareness paired 
comparison technique. 

Situation Awareness 


Independent ANOVA analyses revealed the presentation of CDTI was significant for PF SA-SWORD 
ratings (F(2,30)=216.273, pO.OOOl) and PM SA-SWORD ratings (F(2,30)=227.619, pO.OOOl). 
Post-hoc tests (SNK using a=0.05) showed the same three unique subsets for the PF and PM SA- 
SWORD ratings: 1) No CDTI (lowest SA), 2) CDTI on Moving Map, and 3) CDTI on Moving Map 
and Runway Inset (highest SA). 

Traffic Awareness 

Independent ANOVA analyses revealed the presentation of CDTI was significant for PF traffic 
awareness ratings (F(2,30)=44.740, pO.OOOl) and PM traffic awareness ratings (F(2,30)=5819.857, 
p<0.0001). Post-hoc tests (SNK using a=0.05) showed the same three unique subsets for the PF and 
PM traffic awareness ratings: 1) No CDTI (lowest traffic awareness), 2) CDTI on Moving Map, and 
3) CDTI on Moving Map and Runway Inset (highest traffic awareness). 

4. 5. 3 Unexpected Runway Incursion 

Non-normal runs were injected into the test unbeknownst to the PF and PM. Of the non-normal runs 
one was an unexpected runway incursion scenario, followed by nine runway incursion scenarios 
which were not identified to the flight crew. However, after experiencing the first unexpected runway 
incursion, the crew may have expected another type of incursion or non-normal scenario. These 
incursion scenarios provide an operationally oriented quantification of traffic awareness using SEVS 
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and CDTI. 


The unexpected runway incursion scenario was flown once by each crew. This run was always the 
30 th trial out of 39 trials flown by each crew. 

The runway incursion run was always flown to Runway 22R without TDZ/CL lights in a 10 knot 
headwind and 1400 ft RVR OTW test condition with one of four display configurations 

1) EFVS without CDTI, 

2) EFVS with CDTI on moving map and runway inset, 

3) SVS HDD without CDTI, or 

4) SVS HDD with CDTI on moving map and runway inset. 

The PF could identify the incurring aircraft OTW, with the FFIR on HUD (for EFVS runs), or with 
CDTI (if available HDD). The PM could identify the incurring aircraft OTW, with FFIR on HDD 
(for EFVS runs), or with CDTI (if available HDD). In Figure 20, the FFIR signature from the 
incurring vehicle is shown after the aircraft has turned onto the runway. 



Figure 20. Runway Incursion aircraft as seen on FLIR on PM head-down display. 

Table 34 shows the four display configurations used for the unexpected runway incursion, if the 
incursion was detected, the detection altitude, and PF eye location when the incursion was detected. 
Also included in Table 34 are event description data gathered from video review of the unexpected 
runway incursion runs. 
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Table 34. Unexpected Runway Incursion Run Data. 


Display 

Concept 

CDTI 

Incursion 

Detected 

AGL Altitude 
when 
Incursion 
Detected (ft) 

PF Eye 

Location when 
Incursion 
Detection 

Event Description 

EFVS 

Moving Map + 
Runway Inset 

yes 

26 

OTW 

PF called go-around at 26 ft AGL 

EFVS 

Moving Map + 
Runway Inset 

yes 

230 

OTW 

Using CDTI on runway inset, PM noted another 
aircraft approaching runway at ~340 ft AGL. PM 
called "go around" at 230 ft. 

EFVS 

Moving Map + 
Runway Inset 

no 

N/A 

No data 

Neither pilot saw incurring vehicle until after they 
landed. (No eye tracking data available.) 

EFVS 

None 

yes 

4 

OTW 

PF noticed aircraft at 4 ft AGL and called go- 
around. PM had head in cockpit and didn't see 
incursion. PF felt like he was looking through 
incurring aircraft on the runway. 

EFVS 

None 

yes 

22 

No data 

PM called go-around at about 22 ft AGL. PF saw 
incurring aircraft about the same time. (No eye 
tracking data available.) 

EFVS 

None 

yes 

40 

OTW 

PF saw engines and hit TOGA button. No audible 
recognition of incursion prior to TOGA press. PM 
didn't see incursion. 

SVS PFD 

Moving Map + 
Runway Inset 

yes 

8 

OTW 

Neither pilot had moving map or runway inset in 
visual scan. PM called traffic on runway/go-around 
at 8 ft AGL 

SVS PFD 

Moving Map + 
Runway Inset 

yes 

32 

OTW 

PM called go-around. Noted on CDTI at first, then 
looked away. Finally saw on moving map display. 

SVS PFD 

Moving Map + 
Runway Inset 

yes 

240 

Head-Down 

PM commented on traffic going onto runway at 
270 ft AGL and called go-around at 240 ft AGL 

SVS PFD 

None 

yes 

0 

OTW 

PM called for go-around at 0 ft. 

SVS PFD 

None 

yes 

2 

OTW 

PF saw just before landing at 2 ft AGL and called 
go-around. 

SVS PFD 

None 

yes 

4 

No data 

PM saw incurring aircraft just prior to touchdown. 
Go-around called at 4 ft AGL. (No eye tracking 
data available.) 


The data show that: 

• CDTI was beneficial in only two of six runs (when the incurring vehicle was recognized by 
the crew above 200 ft AGL). 

• For one of the three EFVS runs with CDTI, the crew did not notice the runway incursion until 
landing on it. 

• For nine of 12 runs, crews did not notice incurring aircraft until under 40 ft AGL, with five of 
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them performing a go-around under 20 ft AGL. 

4 . 5 . 4 Expected Runway Incursions 

The “expected” runway incursion runs were always given to the pilots in the second phase of 
experimental testing, immediately following the unexpected runway incursion. These runs were 
flown to runways without TDZ/CL lights (4R, 22L, and 22R) using the EFVS with one of three HDD 
CDTI configurations (none, Moving Map, or Moving Map and Runway Inset). As was expected, the 
crews were more tuned to potential runway incursion events after being exposed to the unexpected 
runway incursion event. As such, all crews noticed the expected runway incursions and performed a 
go-around. However, one crew did not notice the three expected runway incursions until under 30 ft 
AGL, with two of these three runs under 20 ft AGL. On average, the other crews noticed the 
expected runway incursions and performed a go-around at 186 ft AGL (with standard deviation of 61 
ft). 

There were no significant (p>0.05) CDTI effects on incursion detection time from beginning of run 
(mean =75 sec, a=6.6 sec) or detection altitude (mean=71 ft, a=75 ft) for the expected runway 
incursion runs. The mean AGL altitude when incursion was detected was slightly higher (i.e., earlier 
detection) with the Moving Map and Runway Inset CDTI format, however, it was not significantly 
different than Moving Map or No CDTI formats. 
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5 Operational Credit Discussion of Results 

The results of a pilot-in-the-loop simulation of SEVS technologies were presented using an 
operational comparative baseline in the preceding section. 

In the following, these results are discussed, again, using an operational comparative baseline, 
focusing on performance and safety to identify if consideration for operation credit is warranted in the 
following conditions: 

• To enable descent below the published DA/DH on a straight-in instrument approach 
procedure, landing, and roll-out in visibilities as low as 1000 ft RVR by use of an EFVS 
without natural visibility references 

• To enable descent to a DA/DH as low as 150 ft HAT on an ILS approach by use of SVS 

5.1 EFVS Operational Credit 

The EFVS simulation results, in terms of performance and safety, are discussed with respect to the 
following five items to identify if consideration for operation credit is warranted for EFVS. 

1 . Did EFVS operations affect the existing instrument segment? 

For consideration of operational credit, the use of EFVS should “do no harm” in the 
instrument segment. 

2. Did EFVS provide equivalent levels of safety and performance in the visual segment to the 
operational comparative baseline? 

For consideration of operational credit, the use of EFVS should provide equivalent, if not 
better, levels of performance and safety in the visual segment (from DH to 100 ft) as the 
operational comparative baseline. This comparison is identical to the previous certification 
basis used for EFVS under the existing § 91.175 (1) and (m) rule. 

3. Could pilots safely land and roll-out using EFVS? 

For consideration of operational credit, the use of EFVS should provide equivalent, if not 
better, levels of performance and safety in the flare, landing, and roll-out as the operational 
comparative baseline. This comparison expands the previous certification basis under § 
91.175 (1) and (m) rule and if successful, essentially eliminates the requirement for natural 
vision to conduct approach, landing, and roll-out. 

4. Any effect of clutter / obscuration using EFVS? 

The combination of what the pilot can see in the FFIR image and HUD, and what can be seen 
through and around the HUD, must be as safe and effective as the view without the EFVS 
image and HUD. This element primarily, but not exclusively, considers the influence of 
HUD clutter and potential obscuration of the PF’s view out of the window. 

5. Influence of EFVS design factors and off-nominal operations 

Off-nominal performance and non-normal scenarios should not create unacceptable or unsafe 
situations or conditions. Not all possible off-nominal situations were simulated in the test, 
but a few pertinent ones were tested and are reviewed as they may possibly influence the 
acceptability and utility of EFVS and thus, consideration for operation credit. 
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The discussion collapses the comparison between the operational baseline (traditional head-down 
Primary Flight Display without synthetic or enhanced vision) flown in 1800 ft RVR to a 200 ft DH 
against a EFVS HUD evaluated in 1400 ft or 1000 ft RVR conditions (i.e., collapsed across the 1400 
ft and 1000 ft RVR and runway lighting conditions). 

5. 7. 7 EFVS Instrument Segment 

In accordance with accepted FAA approach standards (FAA-S-8081-4D and AC120-29), the 
glideslope tracking (-90 to 95% of runs meeting the standards) and localizer tracking (100% of runs 
meeting the standards) for the baseline and EFVS HUD conditions were equivalent. Using RMS 
deviation performance statistics in the instrument segment (from 1000 ft to DH), the EFVS HUD 
condition provided better glideslope tracking and lower sink rate deviation over the operational 
baseline. 

Equivalent and improved performance for the EFVS HUD was felt to be due to two considerations. 
First, the HUD gives a conformal presentation of flight path and guidance which improves flight 
technical performance in comparison to a minified PFD. Even though the symbology and guidance 
algorithms are the same, the increased sensitivity of the HUD symbolic scaling improves the pilot’s 
ability to track the flight path and maintain a stabilized instrument approach. Secondly, as the aircraft 
descends toward the DH, a pilot using a head-down display tends to look out the forward windscreen 
for the emergence of visual flight references. Although technically in the instrument segment, this 
natural behavior detracts from the pilot’s full attention to flight technical performance. As one pilot 
said, “look up, fly up” noting a natural tendency of a pilot to shallow their flight path when initially 
transitioning to head-up flight conditions. Performance also suffers due to the time spent in visual 
accommodation transitioning between the OTW and head-down instruments. In comparison, the 
EFVS HUD allows nearly simultaneous attention to flight technical performance as well as 
emergence of visual flight references, inherent to the HUD. These performance improvements are 
especially notable since, in EFVS HUD case, worse performance might be expected in sink rate and 
glideslope tracking performance because the EFVS used a lower DH. With a lower DH, the 
increased sensitivity of glideslope should make glideslope tracking more difficult and the emergence 
of enhanced visual references should divide the attention of the pilot-flying. 

For consideration of operational credit, the use of EFVS should “do no harm” in the instrument 
segment. These results show that EFVS clearly does no harm, and in fact, improves the ability of the 
pilot to fly in the instrument segment compared to the operational baseline. 

5.1.2 EFVS Visual Segment 

Flight technical performance in the visual segment is best quantified by the JAR-AWO results, 
identifying the probability of success in the instrument to visual segment. Glideslope tracking must 
be within 1 dot and localizer tracking within 1/3 dot from 300 ft to 100 ft HAT. The results show that 
localizer tracking for the EFVS operational baseline condition (Conventional HDD) and EFVS HUD 
conditions were equivalent. However, glideslope tracking was much better with the EFVS HUD 
conditions than the operational baseline. The probability of success was 95% to 97% for the 
HUD/EFVS condition, but only 29% for the Conventional HDD condition. This performance is also 
reflected in the two go-arounds being conducted by pilots using the Conventional HDD condition 
whereas none of the EFVS HUD approaches ended in a go-around. 

Another way of quantifying FTE within the visual segment is to evaluate how well the aircraft was in 
position to land at 100 ft and upon crossing the threshold. In Figure 21, the aircraft position upon 
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reaching 100 ft HAT is plotted for both the EFVS HUD and Conventional HDD conditions. This 
figure shows the vertical and lateral deviation of the aircraft from the three degree descent path to the 
glidepath intercept point. The figure illustrates the superior vertical positioning when flying the 
EFVS HUD. The vertical deviation is much less, indicating that the aircraft is more closely aligned 
with the glidepath to landing. Lateral deviation of the EFVS HUD configurations is less than the 
operational baseline condition, but not significantly so. 
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Figure 21. Aircraft Position at 100 ft HAT, Conventional HDD vs. EFVS 

In Figure 22, the aircraft position upon crossing the threshold is plotted for both the EFVS HUD and 
Conventional HDD conditions. This figure shows lateral deviation of the aircraft from the runway 
centerline and the altitude deviation from a 50 ft threshold crossing height. Again, the figure 
illustrates the superior vertical positioning when flying the EFVS HUD. The vertical deviation is 
much less, indicating that the aircraft is in a much better position to land within the touchdown zone. 
Also, performance is much more repeatable. Lateral deviation of the EFVS HUD configurations is 
less than the operational baseline condition, but not significantly so. 
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Figure 22. Aircraft Position at the Threshold, Conventional HDD vs. EFVS 

The superior performance with the EFVS HUD condition can largely be traced to the display type 
difference. Although technically in the visual segment, the eye gaze tracking data showed 
statistically significant differences between the Conventional HDD and EFVS HUD conditions. 
Whereas almost no head-up/head-do wn visual transitions were made by pilot’s flying the EFVS HUD 
condition, an average of four head-up-to-head-down (and vice versa) visual transitions were made 
when flying the Conventional HDD condition. Approximately 35% of the time was spent head-down 
by the pilot flying the Conventional HDD condition in the visual segment. This head-down time was 
time well-spent, checking flight instrumentation, but flight technical performance is comparatively 
better when using the HUD. The pilot flying a HUD was able to simultaneously attend to the visual 
references and the instrumentation, without shifting attention. The HDD condition also was impacted 
by a minified flight path and guidance portrayal versus conformal, non-minified HUD flight path and 
guidance. 

For consideration of operational credit, the use of EFVS should provide equivalent, if not better, 
levels of performance and safety in the visual segment (from DH to 100 ft) as the operational 
comparative baseline. This comparison shows that performance in the visual segment is significantly 
better for the EFVS HUD condition than the operational baseline. Further, since the Conventional 
HDD condition still tends to draw the pilot head-down, inside the cockpit, even though they are in the 
visual segment, one can logically argue that the EFVS HUD condition improves the safety of flight 
by increasing the time that a pilot is head-up, looking outside the aircraft. 

5. 1 . 3 EFVS Landing and Roll-Out 

The landing and touchdown data shows no statistically significantly differences in touchdown 
statistics (longitudinal and lateral position, sink rate) between the EFVS HUD and Conventional 
HDD condition. The data showed that there were no wing/empennage strikes of the ground in either 
condition. All touchdowns, for both the EFVS HUD and Conventional HDD conditions, resulted in a 
touchdown within the touchdown zone (200 ft to 2700 ft from the threshold), with the gear within the 
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lateral confines of the runway, and with an acceptable sink rate. 

The eye gaze tracking data showed statistically significant differences in the number of head- 
down/head-up eye transitions in the flare and landing segment. In the flare segment, pilots still had 
on average one head-down/head-up transition when flying with the Conventional HDD condition. (In 
the landing and roll-out segment, the visibility condition triggered changes in the number of head- 
down/head-up eye transitions.) 

For consideration of operational credit, the use of EFVS should provide equivalent, if not better, 
levels of performance and safety in the flare, landing, and roll-out as the operational comparative 
baseline. The simulation results show that landing performance and roll-out performance is 
equivalent to the operational baseline condition. The simulations included headwinds, tailwinds, and 
direct crosswinds up to 15 knots. The EFVS HUD used a flare prompt, but did not include any form 
of flare guidance. 

Two other caveats are important in understanding these data. Overall, the data showed higher sink 
rates at touchdown than what would normally be expected. These sink rates did not differ by display 
configuration. There was a general simulation trend across all experimental variations. Two 
contributing causes were identified. First, the simulation did not include motion. Fixed-base 
simulations have been found to yield higher sink rates at landing than identical simulations conducted 
using motion-base cueing [37]. Second, the simulation used sidestick controllers on an aircraft 
normally equipped with a wheel and column. Fly -by-wire control laws were not implemented to 
improve the handling characteristics of the vehicle. So the combination of unique handling qualities 
and the fact that no real-world training or experience with sidestick controllers on the simulated 
aircraft was available to the flight crews before this experiment contributed to the higher than normal 
sink rates at touchdown. 

5.1.4 EFVS Clutter and Obscuration 

Non-normal runs were injected into the test unbeknownst to the PF and PM. Of the non-normal runs 
one was an unexpected runway incursion scenario, followed by nine expected runway incursion 
scenarios. These incursion scenarios provide an operationally oriented, quantification of traffic 
awareness using SEVS and CDTI. 

The unexpected runway incursion suggests that the use of EFVS improves the PF’s ability to detect 
runway incursions. When the data are collapsed across the CDTI condition, in three out of six runs 
with the EFVS, the PF was the first pilot to detect the runway incursion. In only one out of six runs 
without EFVS (i.e., flying the SVS PFD condition), the PF was the first pilot to detect the runway 
incursion. The EFVS provided an enhanced view of the incurring traffic in the HUD and this 
imagery aided detection. 

The data shows that the EFVS imagery does not clutter or obscure the PF’s view out of the window 
but instead, enhances the PF’s view out of the window. The test did not, however, test this hypothesis 
in weather conditions where the natural outside visibility and the enhanced visibility conditions were 
nearly equivalent. 

5.1.5 EFVS Off-Nominal Operations 

Three off-nominal conditions using the EFVS were given unexpectedly to the flight crew. On two 
runs, the HUD and EFVS failed with the RVR at 700 ft and at 1000 ft upon descending below 50 ft 
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HAT. On another run, the FLIR provided only 700 ft of visibility, instead of the normal 2400 ft 
visibility. The crew’s decision-making process when confronted with these non-normal events while 
flying in low-visibility conditions with the EFVS was assessed through descriptive statistics and 
(where appropriate) ANOVA analyses on run type (normal and non-normal) differences for flight 
performance. 

EFVS and HUD Failure 


Two EFVS failure runs (one in 1000 ft RVR and one in 700 ft RVR) were flown by each crew. The 
crews were not briefed or trained on this event. 

For the 1000 ft RVR EFVS failure condition, 1 1 of the 12 crews made the decision to land despite the 
complete loss of the HUD symbology and FLIR imagery. One crew made the decision to go-around 
at 38 ft AGL. 

For the 700 ft RVR EFVS failure condition, 10 of the 12 crews made the decision to land despite the 
complete loss of the HUD symbology and FLIR imagery. Two crews made the decision to go-around 
at 30 ft and 34 ft AGL for this test condition. 

ANOVA analyses on run type (F(2,41)=l 1.044, p<0.0001) with subsequent SNK post-hoc tests 
revealed that the lateral touchdown performance was significantly worse for the 700 ft RVR EFVS 
Failure runs (mean=-19.5 ft) compared to the 1000 ft RVR EFVS Failure runs (mean=5 ft) and EFVS 
normal runs (mean=-0.3 ft). There were no significant (p>0.5) run type (EFVS normal, 700 ft RVR 
EFVS Fail, 1000 ft RVR EFVS Fail) differences for the other touchdown measures of longitudinal 
position, sink rate, or airspeed. 

The EFVS failures occurred at 50 ft HAT. At this position, the data shows that the aircraft is in 
position to land, having just crossed the runway threshold, on speed and configured for landing due to 
the benefits of EFVS and guidance. When the EFVS HUD failure occurs, the data shows that most 
pilots just continued the operation since, in their opinion, there was sufficient outside visibility to 
safely complete the landing and roll-out. Pilot commentary remarked that even 700 ft RVR was 
sufficient outside visibility to complete the landing and roll-out, but 700 ft RVR was about the limit. 
Any less visibility and they would have been forced to go-around at this low altitude because they 
would not have had sufficient visibility to complete the operation. 

Insufficient Enhanced Flight Visibility to Land 

One insufficient enhanced flight visibility (700 ft RVR) to land run was flown by each crew. This run 
was flown to Runway 22L without TDZ/CL lights in a 7.5 knot right crosswind. 10 of the 12 crews 
made the decision to continue the approach and land even though the EFVS HUD did not provide 
sufficient visual flight references at the DH and 100 ft HAT points. Two of the 12 crews correctly 
followed EFVS crew procedures and initiated a go-around for this test condition. It should be noted 
that approximately 0.5 seconds after the 100 ft altitude aural callout the FLIR did provide sufficient 
visual flight references to complete the approach. This experiment set-up may have unintentionally 
encouraged crews to continue the approach since the FLIR visibility became sufficient to land so 
close to the 100 ft HAT point. 

ANOVA analyses revealed that there were no significant (p>0.05) run type (700 ft RVR FLIR or 
2400 ft RVR FLIR) differences for the touchdown measures of longitudinal position, lateral position, 
sink rate, or airspeed. 
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Off-nominal performance and non-normal scenarios should not create unacceptable or unsafe 
situations or conditions. Not all possible off-nominal situations were simulated in the test, but a few 
pertinent ones were tested and are reviewed as they may possibly influence the acceptability and 
utility of EFVS and thus, consideration for operation credit. 

5.2 EFVS Summary 

The data suggests that operational consideration for the use of EFVS HUD to enable descent below 
the published DA/DH on a straight-in instrument approach procedure, landing, and roll-out in 
visibilities as low as 1000 ft RVR without natural visibility references is warranted. 

5.3 SVS Operational Credit 

The SVS simulation results, in terms of performance and safety, are discussed with respect to the 
following five items to identify if consideration for operation credit is warranted for SVS. 

1 . Did SVS provide equivalent levels of performance and safety in the Instrument Segment 
compared to the Operational Baseline? 

For consideration of operational credit, the use of SVS should provide equivalent, if not 
better, levels of performance and safety in the instrument segment - to a DH of 150 ft - as the 
operational comparative baseline. 

2. Did SVS positively influence the pilot's visual search/runway acquisition at or before 
DA/DH? 

For consideration of operational credit, the use of SVS should positively contribute toward 
the pilot’s ability to transition from the instrument to the visual segment for awareness of the 
landing runway location, positively influencing the identification/verification of the landing 
runway, and acquiring the natural vision landing references. 

3. Did SVS affect performance and safety within the Visual Segment, including the ability to 
safety land and roll-out? 

For consideration of operational credit, the use of SVS should “do no harm” once established 
within the visual segment and for landing and roll-out operations. 

4. Any Effect of Clutter / Obscuration Using SVS? 

In the case of the SV-HUD, the combination of what the pilot can see in the SVS image, and 
what can be seen through and around the HUD, must be as safe and effective as the view 
without the HUD. This performance primarily but not exclusively considers the influence of 
HUD clutter and obscuration on the pilot-flying. 

5. Influence of SVS Design Factors and Off-nominal Operations 

Off-nominal performance conditions and scenarios should not create an unacceptable or 
unsafe situation. Not all possible off-nominal situations were simulated in the test, but a few 
pertinent ones were. 

The discussion collapses the comparison between the operational baseline (HUD without Synthetic or 
Enhanced Vision) flown in 1800 ft RVR to a 150 ft DH against SVS shown on a HUD (collapsed 
across evaluations in 1400 ft or 1000 ft RVR conditions) or on a HDD (collapsed across the 1800 ft 
and 1400 ft RVR). Runway lighting conditions were also collapsed across these conditions. 
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5. 3.1 SVS Instrument Segment 

Using RMS deviation performance statistics in the instrument segment (from 1000 ft to DH), the SVS 
HUD condition showed no statistically significant differences in performance compared to the 
operational baseline (HUD/No SVS). Conversely, the SVS HDD condition showed degraded 
glideslope tracking and higher sink rate deviation compared to the operational baseline. Localizer 
tracking in all conditions was equivalent. When these data are analyzed in accordance with accepted 
FAA approach standards (FAA-S-8081-4D and AC 120-29) which emphasize the maximum 
glideslope and localizer deviations instead of RMS deviation, the glideslope (-90% to 100% of runs 
meeting standards) and localizer (100% of runs meeting standards) tracking were essentially 
equivalent. These results suggest that the statistically significant differences found when analyzing 
RMS deviation may not be operationally relevant, at least to the 200 ft HAT point, used as the lowest 
altitude in AC 120-29. 

As one might logically assume, the eye gaze tracking data showed that the SVS HDD concept had 
significantly more head-down time during the instrument segment. The pilots were head-up greater 
than 94% of the time with the HUD concepts and less than 22% of the time with the SVS HDD 
concepts. The data did show that the pilot- flying transitioned more, between head-up and head-down, 
when flying with the SVS HDD than with the HUD conditions (with and without SVS). But the data 
also shows that when flying with a HUD, the pilots were not exclusively head-up. The pilots 
averaged between three and five head-up/head-down transitions when flying the HUD. This result 
suggests that the pilots when flying the HUD were not exclusively using the HUD, but did transition 
into the cockpit, yet they did not stay inside for long. 

Another way of quantifying FTE within the visual segment is to evaluate how well the aircraft could 
be ‘delivered’ during the instrument segment to the runway. 

In Figure 23, the aircraft position upon reaching 200 ft HAT is plotted for the Conventional HUD and 
the SVS HUD conditions. This figure shows the vertical and lateral deviation of the aircraft from the 
three degree descent path. The figure illustrates almost identical positioning when flying either the 
Conventional HUD or SVS HUD. The presence of SVS neither improved nor degraded aircraft 
positioning to 200 ft HAT. 
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Figure 23. Aircraft Position at 200 ft HAT, Conventional HUD vs. SVS HUD 

In Figure 24, the aircraft position upon reaching 200 ft HAT is plotted for the Conventional HUD and 
the SVS HDD conditions. This figure shows the vertical and lateral deviation of the aircraft from the 
three degree descent path. The figure illustrates almost identical positioning laterally when flying 
either the Conventional HUD or SVS HDD. Slightly better vertical path tracking is shown for the 
Conventional HUD versus the SVS HDD. The minified SVS display and some attention sharing by 
the PF (on occasion) looking outside for emerging visual references are probably the cause of the 
degraded vertical tracking with the SVS HDD condition. 
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Figure 24. Aircraft Position at 200 ft HAT, Conventional HUD vs. SVS HDD 

In Figure 25, the aircraft position upon reaching 100 ft HAT - transitioning through the DH and into 
the visual segment - is plotted for the Conventional HUD and the SVS HUD conditions. This figure 
shows the vertical and lateral deviation of the aircraft from the three degree descent path. The figure 
illustrates almost identical positioning when flying either the Conventional HUD or SVS HUD. The 
presence of SVS neither improved nor degraded aircraft positioning to 100 ft HAT. 

In Figure 26, the aircraft position upon reaching 100 ft HAT is plotted for the Conventional HUD and 
the SVS HDD conditions. This figure shows the vertical and lateral deviation of the aircraft from the 
three degree descent path. The figure illustrates almost identical positioning laterally when flying 
either the Conventional HUD or SVS HDD. Slightly better vertical path tracking is shown for the 
Conventional HUD versus the SVS HDD. The minified SVS display and some attention sharing by 
the PF (on occasion) looking outside for emerging visual references are probably the cause of the 
degraded vertical tracking with the SVS HDD condition. 
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Figure 25. Aircraft Position at 100 ft HAT, Conventional HUD vs. SYS HUD 
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Figure 26. Aircraft Position at 100 ft HAT, Conventional HUD vs. SVS HDD 

For consideration of operational credit, the use of SVS should provide equivalent, if not better, levels 
of performance and safety in the instrument segment - to a DH of 150 ft - as the operational 
comparative baseline. The results show that equivalent performance is provided for the HUD 
condition and, while statistically significant performance differences were shown, acceptable 
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performance when using published FAA approach criteria has been shown for the SVS HDD 
condition tested in this experiment. 

5.3.2 SVS Runway Visual Acquisition 

The eye gaze tracking data was used to examine the influence of SVS on the pilot’s visual 
search/runway acquisition at or before DA/DH. 

The data shows no performance differences with and without SVS for the HUD conditions. 

Intuitively, no visual attention performance differences were expected. A pilot’s attention in both 
cases would principally be focused around the HUD flight path marker and guidance cue information. 
In the absence of errors, the guidance and flight path will be directed toward the synthetic and real 
runways; thus, the pilot’s attention would be directed, by design, in the generally correct 
direction/orientation with and without SVS. 

In comparison of the SVS HDD concept to the operational baseline (HUD/No SVS), the eye gaze 
tracking data is greatly biased by the differences in the display media (HDD vs. HUD) rather than the 
presence or absence of SVS. Pilots are head-down flying the SVS HDD condition approximately 
57% of the time in the instrument-to-visual transition segment versus only approximately 1% of the 
time for the HUD operational baseline. Pilots flying the SVS HDD concepts perform on average 
three to five head-down/head-up transitions, compared to less than one on average for the HUD 
baseline. These data suggest that the display media are extremely influential in this comparison. The 
pilot flying the HUD, with and without SVS, is visually oriented and directed toward the runway. 
Expecting equivalent performance in terms of head-up time, the number of head-up/head-do wn 
transitions, and the direction of eye gaze when transitioning to visual flight references when 
comparing a HUD to a HDD configuration is just not reasonable. 

Instead of using the operational baseline comparison for the HDD, the eye gaze tracking data were 
analyzed for the HDD condition with and without SVS information [38]. When comparing a pilot’s 
first transition to OTW to find the visual references/landing runway, the data show no statistically 
significant differences due to the presence or absence of SVS on the HDDs. However, the data does 
trend toward better performance for SVS (82% of the transitions were in the correct direction to the 
runway vs. 73% correct without SVS). For a full transition to visual flight (the time when the pilot 
goes head-out and stays predominately head-out for landing), the presence of SVS did support a 
better transition. 87% of the time, the pilot using a HDD with SVS correctly looked in the proper 
direction for the runway versus only 66% of the time without SVS. In the instrument-to-visual 
transition segment, no statistically significant differences in head-up time or in head-up/head-down 
transitions were found. 

Performance data were also used to examine the influence of SVS on the pilot’s ability to transition 
from the instrument segment to the visual segment. 

The JAR-AWO approach standards, which spans the 300 to 100 ft HAT range, is appropriate in this 
analysis since it represents the transition from the instrument segment and into the visual segment. 
Under the JAR-AWO criteria, all configurations had 100% probability of successfully meeting the 
within 1/3 dot localizer tracking criteria. The probability of success in meeting the within 1-dot 
glideslope tracking criteria showed display configuration effects. On average, the probability of 
success was 90% for SVS HUD conditions, reducing to 66% for the Conventional HUD condition (no 
SV), and only being -35% for the SVS HDD. These numbers reflect again, the influence of the 
display (HUD vs. HDD) and the improved flight technical performance afforded by the HUD versus 
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the HDD. 


The pilot’s ability to transition from the instrument segment to the visual segment is also manifested 
in the go-around rates seen in the experiment. For the baseline condition (i.e., conventional HUD), in 
1400 ft RVR visibility conditions, one pilot executed a go-around out of 12 total. For the SVD HDD 
condition, one out of 12 pilots executed a go-around with 1800 ft RVR visibility but four executed 
go-arounds with 1400 ft RVR. Similarly, the SVS HUD condition also had four out of 12 pilots 
execute go-arounds with 1400 ft RVR. This data suggests that SVS did not reduce the go-around 
rate, either using a HUD or HDD. In fact, the presence of SVS increased the go-around rate. The 
HDD condition is saddled with the head-down to head-up visual transition problem. In the SVS 
HUD condition, the data suggest that a careful design is required for the SVS HUD since the SV must 
be decluttered to see the natural vision references. Unlike EFVS where the FLIR imagery should 
enhance the OTW view, the SVS may actually obscure the OTW natural view. Although the data 
does not positively identify all conditions, pilots noted this possibility and actively tried to mitigate it. 
Only one SVS HUD design was used and it was not subjected to hundreds of hours of evaluation 
before starting formal data collection. It represents one data point. 

The go-around rate increased to seven out of 12 pilots executing go-arounds with only 1000 ft RVR 
in the SVS HUD condition. The impoverished visual references in this condition clearly triggered a 
high go-around rate since the flight crews did not have sufficient natural vision references to safely 
continue to landing. From a 150 ft DA/DH point, 1000 ft RVR will create a high go-around rate. 

For consideration of operational credit, the use of SVS should positively contribute toward the pilot’s 
ability to transition from the instrument to the visual segment for awareness of the landing runway 
location, positively influencing the identification/verification of the landing runway, and acquiring the 
natural vision landing references. The data shows that SVS on the HUD neither improves nor 
degrades awareness of the direction of the runway. Subjective data from the pilots suggest that the 
terrain and runway shown on the SV gives them better SA of the guidance and time to transition. SV 
does not alter the fundamental problem of head-down to head-up transition using a HDD for an 
instrument approach; however, the data suggests that there is a slight improvement in that transition 
with SV (compared to a HDD without SV). The SV used for the HUD can be a critical issue since 
the SV can obscure, not enhance, the pilot’s view of the natural vision references. De-clutter of the 
raster imagery is critical. 

5.3.3 SVS Visual Segment 

The landing and touchdown data shows no statistically significant differences in touchdown statistics 
(longitudinal and lateral position, sink rate) between the SVS concepts and Conventional HUD 
condition. The data showed that there were no wing/empennage strikes of the ground in either 
condition. 

All touchdowns, for both the SVS HUD and Conventional HUD conditions, resulted in a touchdown 
within the touchdown zone (200 ft to 2700 ft from the threshold), with the gear within the lateral 
confines of the runway, and with an acceptable sink rate. Two touchdowns, for the SVS HDD 
condition, resulted in a touchdown outside/short of the touchdown zone. One occurred just short of 
the 200 ft point and the other was short of the threshold. Since the pilot in these cases did not have a 
HUD, it is not likely that SVS contributed to these adverse landings so much as it was the pilot’s 
inability to land the aircraft visually. 

The eye gaze tracking data showed that pilots with the SVS HDD condition were still head-down 
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approximately 23% of the time in the visual segment. This time is obviously significantly different 
than the 93 to near 100% head-up time for the HUD conditions. In the flare segment, pilots still had 
on average one head-down/head-up transition when flying with the SVS HDD condition. This head- 
up/head-down transition effect appears to be symptomatic of a HDD and is not unique to the presence 
or absence of SVS. Further investigation comparing SVS HDD and Conventional HDD concepts 
reveals SVS decreased the head-down time during the visual segment to on average 25% compared to 
35% head-down with the Conventional HDD concept, with no significant variation in number of 
transitions [38]. Analysis of landing and roll out segments showed no difference between concepts. 
These results suggest the transition behavior is symptomatic of the HDD and not unique to the 
presence of SVS. 

5.3.4 SVS Clutter and Obscuration 

In the case of the SV-HUD, the combination of what the pilot can see in the SVS image, and what can 
be seen through and around the HUD, must be as safe and effective as the view without the HUD. 

This performance primarily, but not exclusively, considers the influence of HUD clutter and 
obscuration on the pilot- flying. 

No quantitative data were taken to specifically test this effect. However, pilot commentary suggests 
that the SV presentation in use on the HUD must be carefully designed to avoid obscuring required 
natural vision landing references. In post-test debriefings, six of the 12 PFs commented that it was 
hard to see the real world (e.g., lights, terrain, etc.) through the SV imagery. Specifically, they said: 

• SV on HUD was hard to look through so you had to decide when you need to turn SV off to 
see the real terrain. However, PF would like SV on HUD over Conventional HUD. 

• SV on HUD had lots of clutter. It was difficult to distinguish between real world and display 
so PF toggled back and forth on displaying SV imagery. 

• SV on HUD is good on initial approach. It was hard to see OTW lights with SV imagery but 
would like SV when taxiing. PF suggested to blank out SV on HUD where you would expect 
to see approach lights 

• SV on HUD is not ready. PF had to declutter SV in close. 

• Decluttered SV on HUD to pick up strobes OTW and then put SV back on. HUD 
transmissivity was an issue regardless with or without stroke symbology. 

• PF could not see lights through SV on HUD so went around most times. 

The SV declutter function was useful in this process (i.e., allowing the pilots to declutter the SV 
presentation on the HUD to have a clear view thru the HUD to see the required visual references), but 
they preferred that the SV design would allow the references to appear without decluttering. For 
instance, SV depiction should be “cut-out” so that the runway, edge lines, threshold, and approach 
lights would not be obscured by the synthetic information. 

Two of the PFs praised the use of SV on the HUD during the post-test debriefings. Specifically, they 
said: 


• With SV on HUD all the way to touchdown, PF could easily have completed landings. SV 
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on HUD is nice for SA and maintaining level flight. 


• SV and FLIR on HUD made OTW visibility not relevant. 

Four of the 12 PFs made no post-test comments on using SV on the HUD. 

In the case of the SVS HDD, no effect of clutter or obscuration was observed. 

5.3.5 SVS Off-Nominal Operations 

Two off-nominal conditions using SVS were flown unexpected to the flight crew. One run included a 
lateral navigation system error scenario and the other included a vertical navigation system error 
scenario. The crew’s decision-making process when confronted with these non-normal events while 
flying in low- visibility conditions with SVS concepts was assessed through descriptive statistics and 
(where appropriate) ANOVA analyses on run type (normal and non-normal) differences for flight 
performance. 

Navigation System Errors 

Two navigation system errors, one lateral and one vertical, were flown by each flight crew while 
using SVS HDD in 1400 ft RVR and flying to a runway without TDZ/CL lights. The crew could 
potentially identify the navigation system error by looking OTW or by crosschecking the ILS 
deviation and SVS scene (minified, head-down). 

Lateral Navigation Error 

Table 35 shows the touchdown performance for the lateral navigation error non-normal runs 
compared to the normal runs flown in the same testing conditions (1400 ft RVR, without TDZ/CL 
lights). 


Table 35. Lateral Navigation System Error Touchdown Performance 



Touchdown Longitudinal 
Position (ft) 

Touchdown Lateral 
Position (ft) 

Touchdown Sink Rate 
(ft/sec) 

Touchdown Airspeed 
(knots) 

-131 

Lat 

Error 

+131 

Lat 

Error 

Norm 

Runs 

-131 

Lat 

Error 

+131 

Lat 

Error 

Norm 

Runs 

-131 

Lat 

Error 

+131 

Lat 

Error 

Norm 

Runs 

-131 

Lat 

Error 

+131 

Lat 

Error 

Norm 

Runs 

Number 
of Go- 
Around 

1 

0 

5 

1 

0 

5 

1 

0 

5 

1 

0 

5 

Number 
of Runs 

5 

6 

43 

5 

6 

43 

5 

6 

43 

5 

6 

43 

Mean 

1222.2 

1106.4 

859.8 

6.9 

4.9 

- 1.6 

- 9.5 

- 7.6 

- 6.8 

127.6 

126.6 

127.2 

Std Dev 

774.4 

434.8 

428.3 

14.5 

20.7 

20.8 

3.5 

2.5 

3.1 

2.4 

1.6 

2.1 

Min 

448.2 

439.2 

- 50.0 

- 7.2 

- 23.8 

- 52.4 

- 13.1 

- 11.7 

- 12.5 

124.5 

124.7 

121.3 

Max 

2396.5 

1521.7 

2146.5 

22.8 

28.0 

50.7 

- 3.7 

- 4.9 

- 0.7 

129.7 

128.3 

130.0 
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ANOVA analyses with run type (normal, +131 ft lateral navigation error, -131 ft lateral navigation 
error) as the main factor indicated no significant (p>0.05) differences for any of the touchdown 
measures (longitudinal position, lateral position, sink rate, or airspeed). 

Video review of the +13 1 ft lateral navigation error runs revealed that: 

• All six runs were taken to a landing 

• Five of the six crews commented on the SVS misalignment 
Video review of the -131 ft lateral navigation error runs revealed that 

• Five out of six runs were taken to a landing 

• Two of the five crews commented on the SVS misalignment 

• For the one go-around, the PF called “go around” at 60 ft AGL. PM commented on the SVS 
misalignment after the run was completed. 

Vertical Navigation Error 

Table 36 shows the touchdown performance for the vertical navigation error non-normal runs 
compared to the normal runs flown in the same testing conditions (1400 ft RVR, without TDZ/CL 
lights). 


Table 36. Vertical Navigation System Error Touchdown Performance 



Touchdown Longitudinal 
Position (ft) 

Touchdown Lateral 
Position (ft) 

Touchdown Sink Rate 
(ft/sec) 

Touchdown Airspeed 
(knots) 

-115 

Vert 

Error 

+115 

Vert 

Error 

Norm 

Runs 

-115 

Vert 

Error 

+115 

Vert 

Error 

Norm 

Runs 

-115 

Vert 

Error 

+115 

Vert 

Error 

Norm 

Runs 

-115 

Vert 

Error 

+115 

Vert 

Error 

Norm 

Runs 

Number 
of Go- 
Around 

2 

1 

5 

2 

1 

5 

2 

1 

5 

2 

1 

5 

Number 
or Runs 

4 

5 

43 

4 

5 

43 

4 

5 

43 

4 

5 

43 

Mean 

999.8 

831.9 

859.8 

15.4 

- 0.9 

- 1.6 

- 8.6 

- 6.8 

- 6.8 

129.3 

125.4 

127.2 

Std Dev 

193.5 

221.2 

428.3 

23.9 

9.7 

20.8 

1.4 

2.3 

3.1 

1.3 

2.5 

2.1 

Min 

710.7 

522.5 

- 50.0 

- 12.3 

- 7.9 

- 52.4 

- 10.5 

- 9.8 

- 12.5 

127.6 

123.0 

121.3 

Max 

1116.1 

1113.1 

2146.5 

36.1 

15.7 

50.7 

- 7.2 

- 3.9 

- 0.7 

130.8 

129.6 

130.0 


ANOVA analyses with run type (normal, +1 15 ft vertical navigation error, -1 15 ft vertical navigation 
error) as the main factor indicated no significant (p>0.05) differences for any of the touchdown 
measures (longitudinal position, lateral position, sink rate, or airspeed). 
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Video review of the +1 15 ft vertical navigation error runs revealed that: 


• Five out of six runs were taken to a landing 

• For the one go-around, the PF did not see the runway environment by minimums and called 
“go around” (i.e., properly followed SVS Crew Procedures). 

Video review of the -1 15 ft vertical navigation error runs revealed that: 

• Four out of six runs were taken to a landing 

• One of the four crews commented on the SVS misalignment (“looks like we are landing short 
on synthetic vision”) 

• For the first go-around, the PM called “go around” at 240 ft AGL 

• For the second go-around, the PF called “go around” at 120 ft AGL because he did not see 
the runway environment (i.e., properly followed SVS Crew Procedures). 

These off-nominal performance conditions and scenarios did not create unacceptable or unsafe 
situations. Not all possible off-nominal situations were simulated in the test, but the few that were 
flown all ended safely. 

However, the absence of data in the form of verbal commentary by the pilots in flying these SV 
depictions with very large navigation errors suggests that the pilots are relying almost exclusively on 
the guidance for flight path tracking until visual flight references are obtained. 

5.4 SVS Summary 

The data suggests that operational consideration for the use of SVS to enable descent to a DA/DH as 
low as 150 ft HAT on an ILS approach by use of SVS may be warranted; however, several issues 
merit consideration in the design and approval of these systems. 

First, HUD implementations of SVS promote head-up attention for the PF and eased the instrument to 
visual transition. SVS HUD was very effective in augmenting HUD-based operations with 
significant SA of the runway and runway environment. However, the potential obscuration of the 
required natural vision references by the synthetic view must be considered in the design. Declutter 
by the PF using a declutter switch was effective but the pilots preferred that the SV did not obscure 
these cues by design. The SV should “cut-out” around these important visual cues. 

Second, HDD implementations of SVS were very effective in augmenting HDD-based operations 
with significant SA of the runway and runway environment. The benefit of SV to improve the 
instrument to visual transition was weakly supported but these implementations are still symptomatic 
of non-HUD flight. Several head-up/head-down visual transitions (with and without SV) were made 
on average by PFs, even as they approach the flare, to check head-down instrumentation. This 
divided attention must be considered in operational approval for low-visibility operations. In this test, 
a transition of control upon reaching visual conditions (i.e., pilot-monitored approaches) was not 
tested. 

Lastly, the non-normal runs suggested significant reliance on the ILS-driven guidance and an absence 

87 



of concern for a mis-matched SV depiction. This result suggests that, while an SA benefit was 
provided by SVS, the guidance is critical. No safety issues were revealed in this test. However, non- 
ILS-based guidance systems with less accuracy and integrity merit significant attention. Also, cross- 
comparison of the ILS-based guidance and non-ILS based SV depictions should be automatically 
made and alerting given in the event of significant differences since the data suggests that pilots 
cannot make this determination reliably. However, the flight crews were not specifically trained to 
perform this detection function. Future work should investigate this training effect. 

5.5 Traffic Awareness as Influenced by CDTI and SEVS 

The traffic awareness data (from SAGAT-like probes and unexpected runway incursion runs) shows 
generally mediocre awareness of traffic and runway incursion detection by pilots. This result 
underscores the importance of automatic flight deck-based conflict detection and resolution work, 
such as Enhanced Traffic Situational Awareness on the Airport Surface with Indications and Alerts 
(SURF IA). 

This work also highlights that three issues should be investigated for improved runway incursion 
detection, to complement SURF IA: 

1. The benefits of training in the use of EFVS for runway incursion should be examined. This test 
did not employ any training. A follow-on test should evaluate if specific training for the PF and 
PM would improve runway incursion detection with EFVS. 

2. The benefits of training in the use of CDTI for runway incursion should be assessed. This test did 
not employ any training in the use of CDTI, specifically for runway incursion detection. The 
function of the CDTI was explained to the crew, but the use of CDTI for clearing the runway was 
not described. A follow-on test should evaluate if specific training for the PF and PM would 
improve runway incursion detection with CDTI. 

3. The use of traffic locator boxes on ego-centric displays, driven by ADS-B information, should be 
investigated. The interaction with EFVS should also be considered in this work. Traffic locator 
boxes were not used in this examination of SEVS technologies; however, they may be useful for 
highlighting and cueing for runway incursion. This benefit must be evaluated against the 
potential negative effect of increased display clutter, confusion, or obscuration. 

5.6 Pilot-Monitoring and SEVS 

Selected runs were made by the PF without the PM being in the simulator. These data allow an 
assessment of the effect of OTW visual cue alerting provided by the pilot monitoring on the 
performance and visual attention for the pilot- flying during approach and landing low visibility 
operations. In addition, these runs collect data for correlation and comparison to a follow-on flight 
test configuration. These runs were not conducted to advocate nor imply the possible acceptance of 
single pilot operations for Part 25 aircraft. In addition, the general applicability of these single pilot 
results with respect to operations and equipage may not necessarily be representative of Part 23 -type 
aircraft and GA operations. The limited applicability to GA is due to the high level of piloting 
experience for the subject pilots and higher flight path stability characteristics of the simulated 
aircraft compared to Part 23 GA aircraft. Table 37 shows the touchdown statistics for the EFVS and 
SVS HDD configurations by crew complement (single pilot, two pilots). 
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Table 37. Touchdown Statistics by Crew Complement 



Single Pilot 
EFVS HUD 

Without TDZ/CL Lights 

Crew 
EFVS HUD 

Without TDZ/CL Lights 

Single Pilot 
SVS HDD 

Without TDZ/CL Lights 

Crew 
SVS HDD 

Without TDZ/CL Lights 


1000 ft RVR/150 ft DH 

1000 ft RVR/150 ft DH 

1400 ft RVR/150 ft DH 

1400 ft RVR/150 ft DH 

Number of 
Go-Around / 
Number of Runs 

0/36 

0/12 

7/36 

1/12 

Touchdown Longitudinal 
Position (ft) 

Mean 

892.7 

798.6 

755.8 

685.0 

Std Dev 

361.8 

358.5 

351.4 

389.8 

Min 

427.3 

430.0 

279.4 

26.8 

Max 

2179.0 

1662.7 

1839.5 

1244.0 

Rating 

Desired 

Desired 

Desired 

Adequate 

Touchdown Lateral 
Position (ft) 

Mean 

-0.7 

-1.2 

1.3 

2.5 

Std Dev 

9.9 

14.7 

15.9 

21.8 

Min 

-21.8 

-30.8 

-21.6 

-33.5 

Max 

30.2 

25.9 

28.1 

50.7 

Rating 

Desired 

Desired 

Desired 

Desired 

Touchdown Sink Rate 
(fps) 

Mean 

-7.1 

-7.7 

-7.4 

-6.9 

Std Dev 

2.2 

3.7 

3.0 

2.8 

Min 

-13.4 

-15.8 

-15.5 

-11.7 

Max 

-2.3 

-3.4 

-1.1 

-2.4 

Rating 

Adequate 

Adequate 

Adequate 

Adequate 


The data reveals that: 

• All EFVS runs resulted in a landing, irrespective of single pilot or crew operations 

• 19% (seven out of 36 runs) of the single pilot SVS HDD runs resulted in a go-around 
compared to 8% (one out of 12) of the crew SVS HDD runs flown in the same test condition 
(without TDZ/CL lights, 1400 ft RVR) 

• On average, both single pilot and crew runs using the EFVS HUD (without TDZ/CL lights, 
1000 ft RVR) and SVS HDD (without TDZ/CL lights, 1400 ft RVR) met desired lateral and 
longitudinal touchdown position criteria 

• Sink rates were higher than expected for both single pilot and crew runs using the EFVS 
HUD and SVS HDD configurations 
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ANOVA analyses for the EFVS HUD runs (without TDZ/CL lights, 1000 ft RVR) revealed no 
significant (p>0.05) crew complement (single, crew) differences for touchdown longitudinal position 
from threshold (mean=869 ft, o= 360 ft), lateral position from centerline (mean=-0.8 ft, g= 1 1 ft), or 
sink rate (mean=-7.2 ft/sec, g= 2.6 ft/sec). Similarly, ANOVA analyses for the SVS HDD runs 
(without TDZ/CL lights, 1400 ft RVR) revealed no significant (p>0.05) crew complement (single, 
crew) differences for touchdown longitudinal position from threshold (mean=736 ft, a=359 ft), lateral 
position from centerline (mean=1.6 ft, a=17 ft), or sink rate (mean=-7.3 ft/sec, a=2.9 ft/sec). 

ANOVA analyses for the EFVS HUD runs (without TDZ/CL lights, 1000 ft RVR) revealed no 
significant differences (p>0.05) in crew complement (single, crew) or CDTI configuration (none, 
Moving Map, or Moving Map and Runway Inset) for PF post-run workload ratings (mean= 3.2). PFs 
rated their workload as being “moderate, easily managed, and having considerable spare time” while 
using the EFVS concepts on approach through landing in visibilities of 1000 ft RVR. 

Similarly, ANOVA analyses for the SVS HDD runs (without TDZ/CL lights, 1400 ft RVR) revealed 
no significant differences (p>0.05) in crew complement (single, crew) or CDTI configuration (none, 
Moving Map, or Moving Map and Runway Inset) for PF post-run workload ratings (mean = 3.7). PFs 
rated their workload as being “busy; challenging but manageable; and having adequate time 
available” while using the SVS HDD concepts on approach through landing in visibilities of 1400 ft 
RVR. 

Eye tracking analysis was also conducted for the head-down configurations to evaluate the influence 
of crew assistance on visual behavior and attention. (Eye tracking behavioral differences for the 
HUD configurations were not expected and were not analyzed.) Surprisingly, no significant 
differences in visual attention were found during the approach and flare segments. Significant 
findings were, however, observed during the landing/roll-out segment, shown in Figure 27, indicating 
single pilots made several more transitions between the HDD and OTW. This is explained by the 
difference in task loading between the two comparisons. Two-crew operations allow for the PF to 
maintain attention OTW with the other crew member providing speed, runway remaining, and turn- 
off information callouts especially using the advanced airport moving map display. These tasks are 
critical in the rollout phase, made particularly more difficult in low visibility operations, and are all 
tasked to the individual pilot in single crew operations. This information is only available head- 
down, requiring the single pilot to transition with increased frequency to retain critical attention OTW 
while at the same time collecting the necessary information from the HDDs. 
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Figure 27. Two-Crew vs. Single Pilot SYS Operational Comparison OTW/HDD Transition Count 
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6 Conclusions 


A fixed-base pilot-in-the-loop simulation test was conducted at NASA Langley Research Center 
which evaluated the use of synthetic vision systems/enhanced flight vision systems (SVS/EFVS) in 
Next Generation Air Transportation System low visibility approach/landing operations at Chicago 
O’Hare environment. Various scenarios tested the potential for EFVS for operations in visibility as 
low as 1000 ft runway visual range (RVR) and SVS to enable lower decision heights or visibilities 
than can currently be flown today. 

Objective results indicate that expanding the portion of the visual segment for which EFVS can be 
used -from decision height (DH) to the runway - in visibilities as low as 1000 ft RVR appears to be 
viable as longitudinal and lateral touchdown performance were excellent. Perhaps more important 
than the landing performance results is that the go-around rate was 0% when flying the EFVS 
concept, regardless of the out-the-window (OTW) visibility level (1000 or 1400 ft RVR) or if 
touchdown zone/center line (TDZ/CL) lights were present or not. The enhanced flight visibility was 
held at approximately 2400 ft. Subjective results also supported the expanded use of EFVS from DH 
to the runway. This concept was rated as having less workload and was ranked as the crew’s 
preferred display concept (over the Conventional and SVS concepts tested) to fly with in low- 
visibility conditions. 

RVR appears to affect lateral touchdown performance in the presence of an EFVS failure (i.e., no 
Head-Up Display (HUD) or Forward Looking InfraRed (FLIR) imagery available), but not 
touchdown longitudinal position or sink rate performance. However, all lateral touchdown positions 
were within 19 feet of centerline in the presence of an EFVS failure. 

Results of PF visual behavior under the EFVS operational concepts showed significant increase in 
head up time and reduced number of head up and head down transitions between HUD and Head- 
Down Display (HDD) vision system locations respectively for all in-flight segments of the approach. 
Particular significance was observed in the visual segment, indicating that pilots flying the 
Conventional HDD condition remained head down 35% of the time even after visual acquisition of 
the approach lighting system, continuing to check guidance and instruments available on the HDD. 

All EFVS runs resulted in a landing, irrespective of single pilot or crew operations. Similarly, no 
significant crew complement (single, crew) differences were found for the touchdown position or sink 
rate performance measures. 

Objective and subjective results indicate that using SVS on a HUD to enable lower decision heights 
and/or lower OTW visibility than are currently flown in today’s National Airspace System appears 
viable. Regardless of OTW visibility level or the absence/presence of TDZ/CL lights, all SVS HUD 
approaches were within the landing box (i.e., between 200 ft to 2700 ft longitudinal distance from 
threshold and within ±58 ft lateral distance of centerline) defined in existing performance-based auto- 
land standards (FAA AC120-28D and JAR AWO) for touchdown longitudinal position and lateral 
position from centerline. However, OTW visiblity impacted the go-around rate with the SVS HUD 
concepts, with nearly twice as many go-arounds being performed in 1000 ft RVR than being 
performed in 1400 ft RVR. SVS HUD operations in visibilities as low as 1400 ft RVR with a 150 ft 
DH appear possible. The SVS HUD go-around rates observed in this experiment and post-test pilot 
commentary indicate that synthetic vision presentation in use on a HUD must be carefully designed to 
avoid obscuring required natural vision landing references. 
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Regardless of OTW visibility level or the absence/presence of TDZ/CL lights, all SVS HDD concepts 
evaluated easily met the desired lateral touchdown criteria defined for this test. However, OTW 
visibility impacted the go-around rate for the SVS HDD concepts, with four times as many go- 
arounds being performed in 1400 ft RVR than being performed in 1800 ft RVR. SVS HDD 
operations in 1800 ft RVR with a 150 ft DH appear promising if TDZ/CL lights are present. 

The presence of large, unannunciated navigation system errors (lateral and vertical) did not affect 
pilot touchdown position or sink rate performance while flying with the SVS HDD concepts. The 
tendency to go-around was less profound with the large lateral navigation system error runs (one go- 
around out of 12 possible approaches) than it was with the large vertical navigation system error runs 
(three go-arounds out of 12 possible approaches). 

Results of Pilot Flying (PF) visual behavior under the SVS operational concepts showed significant 
increase in head up time and reduced number of head up and head down transitions between HUD 
and HDD vision system locations respectively for all segments of flight, including flare and landing 
rollout. During the visual segment of flight, the SVS HDD condition eye tracking results indicate 
pilot visual attention remains inside the flight deck 25% of the time. Pilot visual attention continued 
to transition between the OTW scene and flight instruments and guidance available on the HDD. 
Relative to the Conventional HDD condition, this is a 10% increase in head up time when using SVS 
during the visual segment. There were no significant effects in PF visual behavior observed when 
contrasting SVS and Conventional vision systems on the HUD. 

No significant crew complement (single, crew) differences were found for the touchdown position or 
sink rate performance measures for the SVS HDD concept tested. 

In general, having TDZ/CL lights appears to have aided the pilots in landing closer to the touchdown 
aim point (1000 ft past the runway threshold). 

Pilots reported significant gains in overall SA and traffic awareness when they had cockpit display of 
traffic information (CDTI). However, an unexpected runway incursion was not detected when a crew 
was flying with FLIR imagery on the HUD and CDTI head-down. 

Future research should include motion-based simulation testing for the SVS HUD and HDD concepts 
to assess its impact on approach and landing performance, especially in sink rate control on 
touchdown. 
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8 Appendix A 

8.1 Air Force Flight Test Center (AFFTC) Workload Estimate Scale 



Workload Estimate 

1 

Nothing To Do; No System Demands 

2 

Light Activity; Minimum Demands 

3 

Moderate Activity; Easily Managed; Considerable Spare Time 

4 

Busy; Challenging But Manageable; Adequate Time Available 

5 

Very Busy; Demanding To Manage; Barely Enough Time 

6 

Extremely Busy; Very Difficult; Non-Essential Tasks 
Postponed 

7 

Overloaded; System Unmanageable; Important Tasks 
Undone 
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8.2 Paired Comparison Technique 


Crew Pilot Date 


Paired Comparisons 

Overview: This questionnaire is designed to allow statistical analysis of your subjective 

assessment of situation awareness and traffic awareness for each of the following CDTI display 
configurations you evaluated today. Please look at the pictures of the display concepts when 
making your comparisons. 

Paired Comparison Rating Instructions: Eajrii paired comparison will be listed on the left side 
of the questionnaire. You will be asked to make comparisons based on what you experienced 
during the approach task flown to ORD. 

Situation Awareness: If situation awareness is not equal, indicate the magnitude of the 
difference by marking the appropriate box on the scale to the right of the comparison. 
Die following definition of situational awareness should be used for reference: 

The pilot's awareness and understanding of all factors that, will contribute 
to the safe fying of his/her aircraft under normal and non-normal 
conditions . 

Traffic Awareness: If traffic awareness is not equal, indicate the magnitude of the 
difference by marking the appropriate box on the scale to the right of the comparison. 
Die following definition of traffic awareness should be used for reference: 

Trie pilot's awareness and understanding of significant traffic that will 
affect his/her aircraft under normal and non-normal operating conditions. 

Die following example shows how to make the comparisons. Do not take an excessive amount 
of time on each comparison; your first impression is usually best. However, please feel free to 
correct any comparisons. Also, the data will be checked for consistency; if the results are 
inconsistent, you may be asked to clarify your responses. 


Si 


Situational Awareness 

If not equal, how much more or how much less? 
Barely Substantially 

Display Concept 

Provide &(_V_ more X e qua I'M' less) SA than 


V 

Display Concept 
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Crew Pilot Date 


Cockpit Display of Traffic Information (CDTI) on Surface Map 


SITUATION AWARENESS (SA) 

Display Concept ‘A* 

Provides ( moreX equalX less) SA than 

Display Concept ( B J 
Display Concept ‘A* 

Provides ( moreX equal)( less) SA than 

Display Concept i C f 

Display Concept ( B* 

Prov ides ( moreX equalX less) SA than 

Display Concept ‘C* 


TRAFFIC AWARENESS (TA) 

Display Concept ‘A* 

Provides ( moreX equalX less) TA than 

Display Concept ( B‘ 

Display Concept ‘A* 

Provides ( moreX equalX less) TA than 

Display Concept ( C* 

Display Concept ( B‘ 

Provides ( moreX equalX less) TA than 

Display Concept 


If not equal, how much more or how much less? 
Barely Substantially 


If no| equal how much more or how much less? 
Barelv Substantially' 


See PowerPoint slides for Display Concept Pictures 

Display Concept tc A ? ‘ = No CDTI ora Moving Map or Runway Inset 

Display Concept ^R“ = CDTI on Moving Map Gnly 

Display Concept “C” = CDTI on Moving Map and Runway Inset 
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8.3 Example Traffic Probe for Runway 22R 


Traffic - SA GAT Run Cre w / PF or PNF (PM)/ Date 

Instruction: JWajliclMafJon (?f Ait Traffic Hotding Short of Runway 





22R 
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